Nov 02 17:30:31.911638 ip-10-0-135-5 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 17:30:31.911647 ip-10-0-135-5 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 17:30:31.911654 ip-10-0-135-5 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 17:30:31.911879 ip-10-0-135-5 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 17:30:42.089561 ip-10-0-135-5 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 17:30:42.089604 ip-10-0-135-5 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d025974ea6fc49d482f5779611ee8a04 --
Nov 02 17:32:25.028965 ip-10-0-135-5 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:32:25.444554 ip-10-0-135-5 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:32:25.444554 ip-10-0-135-5 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:32:25.444554 ip-10-0-135-5 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:32:25.444554 ip-10-0-135-5 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:32:25.448087 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.447998    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.448087 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448088    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448093    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448096    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448099    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448102    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448105    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448108    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448111    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448114    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448117    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448120    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448123    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448125    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448128    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448130    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448133    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448135    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448138    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448140    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448143    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.448161 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448145    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448148    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448151    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448153    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448156    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448162    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448165    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448168    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448170    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448173    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448176    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448178    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448183    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448187    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448190    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448193    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448196    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448198    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448201    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.448636 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448203    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448205    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448208    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448210    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448213    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448215    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448217    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448220    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448222    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448225    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448227    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448229    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448232    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448234    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448237    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448240    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448243    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448245    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448248    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448251    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.449079 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448254    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448257    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448259    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448261    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448264    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448266    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448270    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448274    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448277    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448280    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448283    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448286    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448289    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448291    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448294    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448296    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448299    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448301    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448304    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.449560 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448306    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448308    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448311    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448314    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448316    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448318    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448321    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448324    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448327    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448330    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448332    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448335    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448337    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448341    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448344    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448346    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448349    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448351    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448354    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448356    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.450074 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448358    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448361    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448363    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448366    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448371    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448373    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.448375    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450055    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450063    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450067    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450070    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450073    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450076    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450079    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450082    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450085    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450088    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450090    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450093    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.450538 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450096    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450099    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450102    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450104    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450107    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450109    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450112    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450115    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450118    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450120    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450123    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450125    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450129    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450133    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450136    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450140    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450143    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450146    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450149    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.451011 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450152    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450155    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450157    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450160    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450162    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450165    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450167    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450171    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450173    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450175    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450178    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450180    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450184    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450187    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450190    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450192    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450195    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450198    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450200    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.451457 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450203    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450205    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450208    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450211    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450214    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450216    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450218    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450221    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450223    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450226    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450229    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450231    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450234    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450236    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450239    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450242    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450244    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450246    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450249    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450252    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.451928 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450254    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450257    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450259    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450262    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450264    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450267    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450270    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450272    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450275    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450278    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450280    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450283    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450285    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450288    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450291    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450294    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450297    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450299    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450303    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.452441 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450306    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450325    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450328    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450331    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450334    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450336    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450339    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450341    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450345    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450349    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450351    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450354    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450356    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450359    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450361    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450364    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450366    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450897    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450903    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450906    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.452903 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450910    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450912    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450915    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450917    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450920    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450922    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450925    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450928    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450931    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450934    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450936    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450939    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450941    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450944    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450947    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450949    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450952    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450954    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450957    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450959    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.454003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450962    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450964    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450967    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450969    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450971    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450974    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450976    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450979    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450982    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450984    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450987    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450990    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450992    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450995    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450997    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.450999    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451002    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451004    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451008    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.454463 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451012    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451015    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451019    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451022    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451025    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451027    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451030    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451032    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451035    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451037    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451040    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451044    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451048    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451051    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451054    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451057    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451059    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451061    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451064    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.454944 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451066    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451069    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451071    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451074    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451076    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451079    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451082    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451085    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451087    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451090    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451092    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451095    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451097    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451100    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451103    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451105    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451108    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451112    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451114    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.455396 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451117    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451119    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451122    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451124    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451127    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451129    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451132    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451135    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451137    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451139    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451142    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451144    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451147    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451149    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451152    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451154    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451157    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451159    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451163    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451166    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.455833 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451168    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451171    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451173    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451175    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451178    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.451180    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452222    2566 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452237    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452243    2566 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452249    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452263    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452267    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452277    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452281    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452284    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452287    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452291    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452294    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452297    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452300    2566 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452303    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452306    2566 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452309    2566 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:32:25.456309 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452311    2566 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452316    2566 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452319    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452322    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452326    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452332    2566 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452337    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452340    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452343    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452346    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452349    2566 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452352    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452355    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452358    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452361    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452366    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452369    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452372    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452375    2566 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452378    2566 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452382    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452386    2566 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452389    2566 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452392    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:32:25.456857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452396    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452398    2566 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452403    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452406    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452409    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452412    2566 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452415    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452418    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452421    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452424    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452427    2566 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452430    2566 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452433    2566 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452438    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452442    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452446    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452449    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452452    2566 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452455    2566 flags.go:64] FLAG: --help="false"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452458    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452462    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452464    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452467    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:32:25.457491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452471    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452475    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452478    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452481    2566 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452484    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452487    2566 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452491    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452494    2566 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452497    2566 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452499    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452502    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452505    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452508    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452511    2566 flags.go:64] FLAG: --lock-file=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452513    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452517    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452520    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452526    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452529    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452532    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452534    2566 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452537    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452542    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452545    2566 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452548    2566 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:32:25.458045 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452552    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452556    2566 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452560    2566 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452563    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452566    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452569    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452572    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452575    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452578    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452581    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452589    2566 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452592    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452595    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452598    2566 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452601    2566 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452604    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452607    2566 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452610    2566 flags.go:64] FLAG: --port="10250"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452627    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452630    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-055f47dd784e84826"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452633    2566 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452636    2566 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452639    2566 flags.go:64] FLAG: --register-node="true"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452642    2566 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452646    2566 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452648    2566 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:32:25.458721 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452651    2566 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452654    2566 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452658    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452662    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452665    2566 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452668    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452671    2566 flags.go:64] FLAG: --runonce="false"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452674    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452677    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452680    2566 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452682    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452685    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452688    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452693    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452696    2566 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452699    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452703    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452705    2566 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452708    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452711    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452714    2566 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452718    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452723    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452726    2566 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452729    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:32:25.459341 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452734    2566 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452737    2566 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452739    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452742    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452745    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452748    2566 flags.go:64] FLAG: --v="2"
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452752    2566 flags.go:64] FLAG: --version="false"
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452757    2566 flags.go:64] FLAG: --vmodule=""
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452760    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.452764    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452905    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452910    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452913    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452916    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452918    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452921    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452924    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452928    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452932    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452935    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452940    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.460243 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452942    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452945    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452948    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452950    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452953    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452955    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452958    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452960    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452964    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452966    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452969    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452971    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452974    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452976    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452979    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452981    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452984    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452986    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452989    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.461169 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452992    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452996    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.452999    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453002    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453005    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453007    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453010    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453013    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453015    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453018    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453020    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453022    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453026    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453029    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453031    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453034    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453036    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453039    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453042    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453044    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.461989 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453046    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453049    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453052    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453054    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453057    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453059    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453062    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453064    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453067    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453069    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453071    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453074    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453076    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453079    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453082    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453084    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453087    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453089    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453092    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453094    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.462918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453096    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453099    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453101    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453104    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453107    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453110    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453112    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453115    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453117    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453119    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453122    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453124    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453127    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453129    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453132    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453142    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453146    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453149    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453152    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.463845 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453154    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453157    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453160    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453163    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453165    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453167    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453170    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453172    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453175    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453178    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453180    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453183    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453185    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453188    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453190    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453193    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.464536 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453195    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.453225    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453268    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453272    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453276    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453279    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453281    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453284    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453288    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453290    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453293    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453296    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453298    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453302    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453304    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453307    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.464943 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453309    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453312    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453315    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453317    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453320    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453322    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453325    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453328    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453330    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453333    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453335    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453338    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453340    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453342    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453345    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453347    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453350    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453352    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453356    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.465564 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453359    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453361    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453364    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453366    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453370    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453373    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453375    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453378    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453380    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453383    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453385    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453388    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453391    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453394    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453396    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453399    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453402    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453404    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453407    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453409    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.466393 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453411    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453414    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453417    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453421    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453424    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453428    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453431    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453433    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453436    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453439    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453441    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453446    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453448    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453451    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453453    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453457    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453460    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453463    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453465    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.466966 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453468    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453470    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453473    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453475    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453478    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453481    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453483    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453487    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453490    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453493    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453495    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453497    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453500    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453503    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453505    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453508    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453510    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453512    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453515    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.467569 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453517    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453519    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453522    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453524    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453527    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453531    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453534    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453536    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453539    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453541    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453545    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453548    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453550    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453553    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.453556    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.462139    2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.462163    2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.462189    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.462198    2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462285    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.468042 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462293    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462298    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462303    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462307    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462311    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462315    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462320    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462324    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462328    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462333    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462337    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462341    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462346    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462351    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462355    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462359    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462363    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462368    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462374    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.468547 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462381    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462386    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462390    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462395    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462400    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462404    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462409    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462413    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462418    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462427    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462434    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462439    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462444    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462448    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462453    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462457    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462462    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462466    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462470    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.469005 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462475    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462479    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462483    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462487    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462491    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462496    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462500    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462505    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462509    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462513    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462517    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462521    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462525    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462529    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462533    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462538    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462543    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462548    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462552    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462556    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.469459 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462560    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462565    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462570    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462576    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462580    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462584    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462589    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462594    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462598    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462602    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462606    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462610    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462633    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462637    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462641    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462645    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462649    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462652    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462657    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462660    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.469958 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462664    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462668    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462671    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462675    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462679    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462682    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462687    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462690    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462695    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462699    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462703    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462707    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462710    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462715    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462719    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462723    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462728    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462733    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462737    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462741    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.470458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462745    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462749    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462753    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462758    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462761    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462766    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.462770    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.462814    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463050    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463059    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463064    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463069    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463074    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463080    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463084    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463088    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.470973 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463093    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463097    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463101    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463106    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463110    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463115    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463119    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463123    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463128    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463132    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463137    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463141    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463145    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463151    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463155    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463160    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463164    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463168    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463172    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463176    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.471348 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463180    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463184    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463188    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463193    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463197    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463202    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463206    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463210    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463214    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463218    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463222    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463226    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463231    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463236    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463240    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463244    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463248    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463253    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463257    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.471918 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463261    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463265    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463269    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463274    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463278    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463282    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463287    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463296    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463300    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463304    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463308    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463312    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463316    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463320    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463325    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463329    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463333    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463337    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463341    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463345    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.472358 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463349    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463353    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463357    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463361    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463365    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463370    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463374    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463378    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463382    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463387    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463391    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463395    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463399    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463404    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463408    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463413    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463417    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463421    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463426    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463430    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.472851 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463435    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463440    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463445    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463450    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463455    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463460    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463464    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463468    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463473    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463479    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463485    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463490    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463496    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463500    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463505    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463510    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463516    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463522    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.473323 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463527    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.463570    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463642    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463650    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463657    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463661    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463665    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463669    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463673    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463678    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463682    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463686    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463690    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463695    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463699    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463704    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:32:25.473824 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463709    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463713    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463717    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463721    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463725    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463730    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463734    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463738    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463742    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463746    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463751    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463755    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463758    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463762    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463767    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463771    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463779    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463784    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463789    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463795    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:32:25.474436 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463800    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463804    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463808    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463812    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463817    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463821    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463826    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463830    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463834    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463838    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463842    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463846    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463851    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463855    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463859    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463862    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463866    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463870    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463874    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:32:25.475291 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463879    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463883    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463887    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463891    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463896    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463900    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463904    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463908    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463912    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463919    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463923    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463927    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463933    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463937    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463941    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463946    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463950    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463954    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463959    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463963    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:32:25.476122 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463967    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463975    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463980    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463984    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463989    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463993    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.463997    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464002    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464008    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464014    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464019    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464024    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464028    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464032    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464036    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464041    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464045    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464049    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464054    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:32:25.476971 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464058    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464062    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464069    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464073    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464078    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464084    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464088    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464092    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464097    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464101    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464105    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464109    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464113    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:25.464119    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.464239    2566 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.470516    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.471563    2566 server.go:1014] "Starting client certificate rotation"
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.471640    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:32:25.477427 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.472435    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:32:25.477866 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.473820    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:32:25.477972 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.477957    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:32:25.490510 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.490484    2566 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:32:25.494512 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.494496    2566 log.go:25] "Validated CRI v1 image API"
Nov 02 17:32:25.495314 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.495286    2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:32:25.499608 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.499588    2566 fs.go:135] Filesystem UUIDs: map[1ea59125-eb7b-49a5-8301-b04cf2db486b:/dev/nvme0n1p4 77583fc4-01b8-4c9e-a019-1735d6f8b2c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 02 17:32:25.499672 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.499608    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 17:32:25.504371 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.504353    2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:32:25.505377 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.505242    2566 manager.go:217] Machine: {Timestamp:2026-11-02 17:32:25.503692096 +0000 UTC m=+0.370536789 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3016299 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249ff89dfae4184363746ec6d8bda0 SystemUUID:ec249ff8-9dfa-e418-4363-746ec6d8bda0 BootID:d025974e-a6fc-49d4-82f5-779611ee8a04 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:a4:45:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:a4:45:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:10:99:b6:b8:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:32:25.505995 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.505985    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:32:25.506144 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.506131    2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:32:25.509631 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.509589    2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:32:25.509824 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.509640    2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-5.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:32:25.509875 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.509827    2566 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:32:25.509875 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.509835    2566 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:32:25.509927 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.509876    2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:32:25.509927 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.509893    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:32:25.511088 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.511076    2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:32:25.515890 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.515875    2566 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:32:25.516031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516022    2566 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:32:25.516075 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516066    2566 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:32:25.516103 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516092    2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:32:25.516137 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516110    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:32:25.516137 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516125    2566 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:32:25.516193 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516137    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:32:25.516675 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.516658    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:32:25.520916 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.520887    2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:32:25.528044 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.528026    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:32:25.528984 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.528971    2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:32:25.529023 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529003    2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:32:25.529171 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529160    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:32:25.529216 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529177    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:32:25.529216 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529183    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:32:25.529216 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529189    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:32:25.529216 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529194    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:32:25.529216 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529202    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:32:25.529999 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.529991    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:32:25.530037 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530001    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:32:25.530037 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530009    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:32:25.530037 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530015    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:32:25.530037 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530023    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:32:25.530037 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530033    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:32:25.530752 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530743    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:32:25.530752 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.530753    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:32:25.534887 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.534874    2566 server.go:1263] "Started kubelet"
Nov 02 17:32:25.534997 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.534927    2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:32:25.535054 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.534961    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:32:25.535054 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.535037    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:32:25.535703 ip-10-0-135-5 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:32:25.536510 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.536493    2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:32:25.537094 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.537079    2566 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:32:25.540327 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.540284    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-5.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 17:32:25.540430 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.540330    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 17:32:25.540430 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.540365    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-5.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 17:32:25.541460 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.540362    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-5.us-east-2.compute.internal.18e44ac144457230  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-5.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-5.us-east-2.compute.internal,},FirstTimestamp:2026-11-02 17:32:25.534845488 +0000 UTC m=+0.401690183,LastTimestamp:2026-11-02 17:32:25.534845488 +0000 UTC m=+0.401690183,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-5.us-east-2.compute.internal,}"
Nov 02 17:32:25.544281 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.544257    2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 17:32:25.546097 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.546007    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:32:25.546629 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.546598    2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:32:25.546833 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.546815    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:25.547405 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547384    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:32:25.547405 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547385    2566 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:32:25.547552 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547413    2566 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:32:25.547552 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547496    2566 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:32:25.547552 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547507    2566 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:32:25.547764 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547746    2566 factory.go:55] Registering systemd factory
Nov 02 17:32:25.547818 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.547769    2566 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:32:25.548018 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.548004    2566 factory.go:153] Registering CRI-O factory
Nov 02 17:32:25.548018 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.548020    2566 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:32:25.548135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.548073    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:32:25.548135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.548087    2566 factory.go:103] Registering Raw factory
Nov 02 17:32:25.548135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.548097    2566 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:32:25.548337 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.548321    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4zbvd"
Nov 02 17:32:25.549235 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.549178    2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-5.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 17:32:25.549359 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.549292    2566 manager.go:319] Starting recovery of all containers
Nov 02 17:32:25.555766 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.555739    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4zbvd"
Nov 02 17:32:25.561424 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.561366    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 17:32:25.562908 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.562890    2566 manager.go:324] Recovery completed
Nov 02 17:32:25.566812 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.566799    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.568754 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.568736    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.568824 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.568773    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.568824 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.568786    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.569503 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.569486    2566 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:32:25.569503 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.569499    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.569601 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.569502    2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:32:25.569601 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.569571    2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:32:25.570224 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.570209    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.570269 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.570241    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.570269 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.570258    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.571588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.571575    2566 policy_none.go:50] "Start"
Nov 02 17:32:25.571671 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.571592    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:32:25.571671 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.571607    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:32:25.573634 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.573608    2566 policy_none.go:44] "Start"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.615306    2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.619871    2566 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.619899    2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.619908    2566 server.go:87] "Starting device plugin registration server"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.620198    2566 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.620210    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.620339    2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.620502    2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.620517    2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.620942    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.621095    2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.621151    2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.622430    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.622455    2566 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.622477    2566 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.622518    2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 17:32:25.627226 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.626222    2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:32:25.721321 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.721243    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.722331 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.722312    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.722456 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.722344    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.722456 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.722354    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.722456 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.722380    2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.722610 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.722593    2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"]
Nov 02 17:32:25.722685 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.722674    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.723392 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.723375    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.723456 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.723403    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.723456 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.723414    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.725659 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.725646    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.725777 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.725762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.725834 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.725800    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.726359 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.726344    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.726449 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.726364    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.726449 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.726377    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.726449 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.726385    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.726449 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.726391    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.726656 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.726394    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.728662 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.728646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.728739 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.728673    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:32:25.729526 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.729510    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:32:25.729608 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.729541    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:32:25.729608 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.729556    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:32:25.733413 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.733396    2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.733502 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.733423    2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-5.us-east-2.compute.internal\": node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:25.748271 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.748248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4cabb9f1887dd6d87ae5b4fba788119d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"4cabb9f1887dd6d87ae5b4fba788119d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.748400 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.748286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.748400 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.748320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.756271 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.756253    2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.759012 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.758995    2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.759652 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.759637    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:25.849343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.849298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4cabb9f1887dd6d87ae5b4fba788119d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"4cabb9f1887dd6d87ae5b4fba788119d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.849540 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.849350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.849540 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.849397    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4cabb9f1887dd6d87ae5b4fba788119d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"4cabb9f1887dd6d87ae5b4fba788119d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.849540 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.849412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.849540 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.849480    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.849540 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:25.849513    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:25.860423 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.860396    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:25.960956 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:25.960924    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:26.061791 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.061736    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:26.061791 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.061743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:26.064933 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.064913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:26.162826 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.162789    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:26.263333 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.263293    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:26.363905 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.363822    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:32:26.462860 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.462823    2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:32:26.470975 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.470956    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 17:32:26.471125 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.471095    2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:32:26.471184 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.471135    2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:32:26.517282 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.517261    2566 apiserver.go:52] "Watching apiserver"
Nov 02 17:32:26.533690 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.533660    2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:32:26.533926 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.533911    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-fh7z4","openshift-dns/node-resolver-dzlmp","openshift-ovn-kubernetes/ovnkube-node-pswgg","kube-system/global-pull-secret-syncer-b26bx","openshift-multus/multus-additional-cni-plugins-vng2x","openshift-multus/network-metrics-daemon-bq652","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2","openshift-image-registry/node-ca-dv585","openshift-multus/multus-7lh5w","openshift-network-diagnostics/network-check-target-pp7tz","openshift-network-operator/iptables-alerter-msvsr","openshift-cluster-node-tuning-operator/tuned-m9pzk"]
Nov 02 17:32:26.536649 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.536628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.538709 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.538690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.540216 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.540194    2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:32:26.540320 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.540310    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:32:26.540541 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.540524    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:32:26.540682 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.540663    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zg2g5\""
Nov 02 17:32:26.541257 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.541218    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.542365 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.542342    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.542467 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.542404    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjf7q\""
Nov 02 17:32:26.542467 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.542429    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:32:26.542672 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.542652    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:32:26.542784 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.542685    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.542784 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.542724    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:32:26.543534 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.543516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.544253 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.544236    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:32:26.544327 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.544265    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.544362 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.544346    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.544665 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.544648    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl4vs\""
Nov 02 17:32:26.545664 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.545645    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.546090 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.546071    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 17:32:26.546550 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.546537    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkfrw\""
Nov 02 17:32:26.546875 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.546862    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:32:26.547685 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.547672    2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:26.547784 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.547772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.548394 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.548378    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:32:26.549197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.549178    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.549316 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.549264    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k78hz\""
Nov 02 17:32:26.549409 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.549394    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.550074 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.550057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.551374 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.551359    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.552353 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.552337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.552423 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.552391    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:26.553483 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.553558 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-log-socket\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553558 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-env-overrides\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553558 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-systemd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553759 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-ovn\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553759 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553759 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d806b75d-88ca-4e51-94c3-cd45fd8888db-agent-certs\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.553759 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-system-cni-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-cnibin\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tjt\" (UniqueName: \"kubernetes.io/projected/032f9f33-d723-46ad-9d89-45956596eec5-kube-api-access-q6tjt\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-systemd-units\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-bin\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-script-lib\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.553952 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d806b75d-88ca-4e51-94c3-cd45fd8888db-konnectivity-ca\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-multus\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.553997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-etc-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-node-log\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-registration-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554068    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-device-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42w6\" (UniqueName: \"kubernetes.io/projected/b90ebafb-a030-43ce-a32a-edc04ce61c09-kube-api-access-t42w6\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-netns\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-kubelet\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-kubelet\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.554197 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-host-slash\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-socket-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-binary-copy\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-socket-dir-parent\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-iptables-alerter-script\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-hostroot\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032f9f33-d723-46ad-9d89-45956596eec5-tmp-dir\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-slash\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-system-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-cnibin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-cni-binary-copy\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.554529 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-conf-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-netns\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-var-lib-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.554659    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554727    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/269ca06c-b59b-496f-81aa-a08c525d0098-ovn-node-metrics-cert\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-bin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-etc-kubernetes\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-multus-certs\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.554960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-multus-daemon-config\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555079 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89ld7\" (UniqueName: \"kubernetes.io/projected/269ca06c-b59b-496f-81aa-a08c525d0098-kube-api-access-89ld7\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47sj4\" (UniqueName: \"kubernetes.io/projected/c58d6490-3f26-48cc-95eb-440097888b15-kube-api-access-47sj4\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-netd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmqbq\" (UniqueName: \"kubernetes.io/projected/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-kube-api-access-mmqbq\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-sys-fs\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-k8s-cni-cncf-io\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zcz\" (UniqueName: \"kubernetes.io/projected/505743f8-9737-44a8-987b-62f5d35212d3-kube-api-access-87zcz\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032f9f33-d723-46ad-9d89-45956596eec5-hosts-file\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.555516 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.555271    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-config\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.556878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.556863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.558082 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.558065    2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:32:26.559190 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.559178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:26.559260 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.559223    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:26.561278 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.561264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.561459 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.561441    2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:32:26.561544 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.561530    2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:32:26.561752 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.561733    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"]
Nov 02 17:32:26.561819 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.561738    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:27:26 +0000 UTC" deadline="2028-03-28 14:01:34.564574395 +0000 UTC"
Nov 02 17:32:26.561819 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.561785    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12284h29m8.002792109s"
Nov 02 17:32:26.570372 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.570352    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"]
Nov 02 17:32:26.570452 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.570392    2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:32:26.580933 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.580914    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bdbds"
Nov 02 17:32:26.590231 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.590198    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bdbds"
Nov 02 17:32:26.640303 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.640280    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.640459 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.640433    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:32:26.640886 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.640869    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.640973 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.640887    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.640973 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.640895    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.641040 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641007    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.641103 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641087    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:32:26.641198 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641115    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbqnj\""
Nov 02 17:32:26.641647 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641629    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:32:26.641724 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641631    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:32:26.641724 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641660    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:32:26.641724 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.641698    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:32:26.642412 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.642395    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dtld9\""
Nov 02 17:32:26.642412 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.642402    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xjkc9\""
Nov 02 17:32:26.642550 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.642442    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:32:26.642550 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.642473    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttfxd\""
Nov 02 17:32:26.648217 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.648203    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.655971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sj4\" (UniqueName: \"kubernetes.io/projected/c58d6490-3f26-48cc-95eb-440097888b15-kube-api-access-47sj4\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-netd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqbq\" (UniqueName: \"kubernetes.io/projected/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-kube-api-access-mmqbq\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-var-lib-kubelet\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-dbus\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-sys-fs\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-k8s-cni-cncf-io\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zcz\" (UniqueName: \"kubernetes.io/projected/505743f8-9737-44a8-987b-62f5d35212d3-kube-api-access-87zcz\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032f9f33-d723-46ad-9d89-45956596eec5-hosts-file\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-config\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656297    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysconfig\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656342    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-tmp\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-log-socket\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.658905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-env-overrides\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncq4r\" (UniqueName: \"kubernetes.io/projected/64d7830f-395c-4694-a61c-c25f1c0adc5a-kube-api-access-ncq4r\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-systemd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-tuned\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-ovn\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d806b75d-88ca-4e51-94c3-cd45fd8888db-agent-certs\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-system-cni-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-cnibin\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tjt\" (UniqueName: \"kubernetes.io/projected/032f9f33-d723-46ad-9d89-45956596eec5-kube-api-access-q6tjt\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-systemd-units\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-bin\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-script-lib\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-systemd\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.660031 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-host\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d806b75d-88ca-4e51-94c3-cd45fd8888db-konnectivity-ca\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-multus\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-etc-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-node-log\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-kubelet-config\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-registration-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-device-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42w6\" (UniqueName: \"kubernetes.io/projected/b90ebafb-a030-43ce-a32a-edc04ce61c09-kube-api-access-t42w6\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-netns\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-kubelet\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-kubelet\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-host-slash\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.656992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zzf\" (UniqueName: \"kubernetes.io/projected/7e1fc379-cd57-4b14-be52-b3975a5f6514-kube-api-access-56zzf\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-socket-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-binary-copy\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-kubernetes\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.660840 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-run\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657141    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-socket-dir-parent\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-iptables-alerter-script\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-hostroot\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032f9f33-d723-46ad-9d89-45956596eec5-tmp-dir\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-slash\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-conf\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-system-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-cnibin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e1fc379-cd57-4b14-be52-b3975a5f6514-serviceca\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-modprobe-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-cni-binary-copy\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-conf-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-netns\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-var-lib-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.661588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e1fc379-cd57-4b14-be52-b3975a5f6514-host\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-sys\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/269ca06c-b59b-496f-81aa-a08c525d0098-ovn-node-metrics-cert\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-bin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-etc-kubernetes\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-multus-certs\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-multus-daemon-config\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89ld7\" (UniqueName: \"kubernetes.io/projected/269ca06c-b59b-496f-81aa-a08c525d0098-kube-api-access-89ld7\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.657973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhxr\" (UniqueName: \"kubernetes.io/projected/fd30ffd8-a9b2-46a3-b057-e25e98540942-kube-api-access-rzhxr\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658035    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-registration-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658056    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-device-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-lib-modules\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658161    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-conf-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.662343 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658200    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-netns\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658237    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-var-lib-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658303    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-host-slash\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658412    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658431    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-socket-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658499    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-sys-fs\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658511    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-socket-dir-parent\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658543    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-k8s-cni-cncf-io\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.658660    2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659078    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-iptables-alerter-script\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659150    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-multus\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659198    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032f9f33-d723-46ad-9d89-45956596eec5-hosts-file\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659214    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-etc-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659262    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-node-log\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659376    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-hostroot\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659687    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032f9f33-d723-46ad-9d89-45956596eec5-tmp-dir\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659744    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-slash\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659769    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-config\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663039 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659841    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.659881    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-log-socket\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660289    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-env-overrides\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660344    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-systemd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660413    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-ovn\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660450    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660500    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660558    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-bin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-etc-kubernetes\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660651    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-multus-certs\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660712    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660772    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-system-cni-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660800    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.660814    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-cnibin\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661246    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-bin\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661290    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.663857 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661413    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-cni-binary-copy\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661467    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-netd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661634    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-cnibin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661733    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-system-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661742    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-multus-daemon-config\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661755    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-script-lib\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661769    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-kubelet\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661803    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-systemd-units\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661849    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-kubelet\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.661884    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-netns\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.662378    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-binary-copy\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.663434    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/269ca06c-b59b-496f-81aa-a08c525d0098-ovn-node-metrics-cert\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.664576 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.663544    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d806b75d-88ca-4e51-94c3-cd45fd8888db-konnectivity-ca\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.665297 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.665280    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d806b75d-88ca-4e51-94c3-cd45fd8888db-agent-certs\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.668488 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.668453    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42w6\" (UniqueName: \"kubernetes.io/projected/b90ebafb-a030-43ce-a32a-edc04ce61c09-kube-api-access-t42w6\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.669093 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.668858    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sj4\" (UniqueName: \"kubernetes.io/projected/c58d6490-3f26-48cc-95eb-440097888b15-kube-api-access-47sj4\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.669396 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.669351    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89ld7\" (UniqueName: \"kubernetes.io/projected/269ca06c-b59b-496f-81aa-a08c525d0098-kube-api-access-89ld7\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.669488 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.669473    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zcz\" (UniqueName: \"kubernetes.io/projected/505743f8-9737-44a8-987b-62f5d35212d3-kube-api-access-87zcz\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.670332 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.670313    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqbq\" (UniqueName: \"kubernetes.io/projected/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-kube-api-access-mmqbq\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.671192 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.671175    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tjt\" (UniqueName: \"kubernetes.io/projected/032f9f33-d723-46ad-9d89-45956596eec5-kube-api-access-q6tjt\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.679488 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.679468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:26.758878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.758847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-conf\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.758878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.758880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e1fc379-cd57-4b14-be52-b3975a5f6514-serviceca\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.759089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.758898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-modprobe-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759010    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-conf\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759013    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e1fc379-cd57-4b14-be52-b3975a5f6514-host\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.759089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759042    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e1fc379-cd57-4b14-be52-b3975a5f6514-host\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.759089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-sys\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759101    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-modprobe-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhxr\" (UniqueName: \"kubernetes.io/projected/fd30ffd8-a9b2-46a3-b057-e25e98540942-kube-api-access-rzhxr\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-lib-modules\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.759202    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-var-lib-kubelet\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.759241    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-dbus\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.759292    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret podName:5188d247-bf34-437c-bde7-0abc73c312f5 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:27.259249655 +0000 UTC m=+2.126094356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret") pod "global-pull-secret-syncer-b26bx" (UID: "5188d247-bf34-437c-bde7-0abc73c312f5") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-dbus\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.759349    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:27.259328715 +0000 UTC m=+2.126173402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759358    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-var-lib-kubelet\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.759375 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759144    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-sys\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759392    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e1fc379-cd57-4b14-be52-b3975a5f6514-serviceca\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysconfig\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-tmp\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759478    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-lib-modules\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759524    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysconfig\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncq4r\" (UniqueName: \"kubernetes.io/projected/64d7830f-395c-4694-a61c-c25f1c0adc5a-kube-api-access-ncq4r\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-tuned\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759634    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-systemd\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-host\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-kubelet-config\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759721    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-systemd\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zzf\" (UniqueName: \"kubernetes.io/projected/7e1fc379-cd57-4b14-be52-b3975a5f6514-kube-api-access-56zzf\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759767    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-host\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-kubernetes\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760022 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-run\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760884 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759812    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-kubelet-config\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:26.760884 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759855    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-kubernetes\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.760884 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.759857    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-run\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.761932 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.761898    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-tmp\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.761932 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.761928    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-tuned\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.767682 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.767610    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:32:26.767682 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.767684    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:32:26.767792 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.767695    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hwsfg for pod openshift-network-diagnostics/network-check-target-pp7tz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:26.767792 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:26.767756    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg podName:812e2a84-04de-4c0e-b8f1-f956381cb3fd nodeName:}" failed. No retries permitted until 2026-11-02 17:32:27.267741068 +0000 UTC m=+2.134585768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hwsfg" (UniqueName: "kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg") pod "network-check-target-pp7tz" (UID: "812e2a84-04de-4c0e-b8f1-f956381cb3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:26.768786 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.768767    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhxr\" (UniqueName: \"kubernetes.io/projected/fd30ffd8-a9b2-46a3-b057-e25e98540942-kube-api-access-rzhxr\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:26.770435 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.770415    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zzf\" (UniqueName: \"kubernetes.io/projected/7e1fc379-cd57-4b14-be52-b3975a5f6514-kube-api-access-56zzf\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:26.771402 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.771384    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncq4r\" (UniqueName: \"kubernetes.io/projected/64d7830f-395c-4694-a61c-c25f1c0adc5a-kube-api-access-ncq4r\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:26.777963 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.777937    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d0975639c634403c0017e572800bfa.slice/crio-de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e WatchSource:0}: Error finding container de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e: Status 404 returned error can't find the container with id de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e
Nov 02 17:32:26.782460 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.782439    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cabb9f1887dd6d87ae5b4fba788119d.slice/crio-7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f WatchSource:0}: Error finding container 7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f: Status 404 returned error can't find the container with id 7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f
Nov 02 17:32:26.783042 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.783030    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:32:26.869421 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.869340    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:26.878753 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.878709    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd806b75d_88ca_4e51_94c3_cd45fd8888db.slice/crio-8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb WatchSource:0}: Error finding container 8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb: Status 404 returned error can't find the container with id 8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb
Nov 02 17:32:26.895956 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.895930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:32:26.902865 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.902846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:32:26.905209 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.905177    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58d6490_3f26_48cc_95eb_440097888b15.slice/crio-83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017 WatchSource:0}: Error finding container 83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017: Status 404 returned error can't find the container with id 83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017
Nov 02 17:32:26.911041 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.911018    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90ebafb_a030_43ce_a32a_edc04ce61c09.slice/crio-e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668 WatchSource:0}: Error finding container e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668: Status 404 returned error can't find the container with id e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668
Nov 02 17:32:26.915890 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.915869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7lh5w"
Nov 02 17:32:26.925873 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.925854    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505743f8_9737_44a8_987b_62f5d35212d3.slice/crio-84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66 WatchSource:0}: Error finding container 84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66: Status 404 returned error can't find the container with id 84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66
Nov 02 17:32:26.933110 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.933091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:32:26.937929 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:26.937916    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:32:26.941489 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.941460    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e2dfb2_b24d_4af8_ad36_7c4532ee90ce.slice/crio-d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac WatchSource:0}: Error finding container d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac: Status 404 returned error can't find the container with id d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac
Nov 02 17:32:26.945904 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:26.945881    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032f9f33_d723_46ad_9d89_45956596eec5.slice/crio-80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a WatchSource:0}: Error finding container 80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a: Status 404 returned error can't find the container with id 80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a
Nov 02 17:32:28.143596 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.143571    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:32:28.149636 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.149610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:32:28.153066 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:28.153045    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1fc379_cd57_4b14_be52_b3975a5f6514.slice/crio-17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0 WatchSource:0}: Error finding container 17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0: Status 404 returned error can't find the container with id 17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0
Nov 02 17:32:28.157855 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:28.157818    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d7830f_395c_4694_a61c_c25f1c0adc5a.slice/crio-24c880010b914afc7edcce914d182aca9278f2d41eda292938760debd5a83897 WatchSource:0}: Error finding container 24c880010b914afc7edcce914d182aca9278f2d41eda292938760debd5a83897: Status 404 returned error can't find the container with id 24c880010b914afc7edcce914d182aca9278f2d41eda292938760debd5a83897
Nov 02 17:32:28.240545 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.240520    2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:32:28.302359 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:28.302331    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269ca06c_b59b_496f_81aa_a08c525d0098.slice/crio-182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a WatchSource:0}: Error finding container 182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a: Status 404 returned error can't find the container with id 182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a
Nov 02 17:32:28.404386 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.404356    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:28.404547 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.404394    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:28.404547 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.404505    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:28.404547 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.404507    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:28.404643 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.404557    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:29.404544647 +0000 UTC m=+3.131166431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:28.404643 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.404569    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret podName:5188d247-bf34-437c-bde7-0abc73c312f5 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:29.404563409 +0000 UTC m=+3.131185192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret") pod "global-pull-secret-syncer-b26bx" (UID: "5188d247-bf34-437c-bde7-0abc73c312f5") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:28.505213 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.505173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:28.505389 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.505329    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:32:28.505389 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.505367    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:32:28.505389 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.505383    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hwsfg for pod openshift-network-diagnostics/network-check-target-pp7tz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:28.505576 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.505442    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg podName:812e2a84-04de-4c0e-b8f1-f956381cb3fd nodeName:}" failed. No retries permitted until 2026-11-02 17:32:29.505424449 +0000 UTC m=+3.232046242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwsfg" (UniqueName: "kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg") pod "network-check-target-pp7tz" (UID: "812e2a84-04de-4c0e-b8f1-f956381cb3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:28.737140 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.737037    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:27 +0000 UTC" deadline="2028-07-10 06:33:16.22617569 +0000 UTC"
Nov 02 17:32:28.737140 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.737090    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14773h0m47.489087849s"
Nov 02 17:32:28.763709 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.763670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:28.763897 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:28.763822    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:28.784124 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.784059    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzlmp" event={"ID":"032f9f33-d723-46ad-9d89-45956596eec5","Type":"ContainerStarted","Data":"80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a"}
Nov 02 17:32:28.807182 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.807141    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msvsr" event={"ID":"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce","Type":"ContainerStarted","Data":"d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac"}
Nov 02 17:32:28.817084 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.817047    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fh7z4" event={"ID":"d806b75d-88ca-4e51-94c3-cd45fd8888db","Type":"ContainerStarted","Data":"8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb"}
Nov 02 17:32:28.821533 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.821505    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"4cabb9f1887dd6d87ae5b4fba788119d","Type":"ContainerStarted","Data":"7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f"}
Nov 02 17:32:28.836419 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.836372    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"22d0975639c634403c0017e572800bfa","Type":"ContainerStarted","Data":"de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e"}
Nov 02 17:32:28.846884 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.846832    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7lh5w" event={"ID":"505743f8-9737-44a8-987b-62f5d35212d3","Type":"ContainerStarted","Data":"84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66"}
Nov 02 17:32:28.848470 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.848381    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2" event={"ID":"b90ebafb-a030-43ce-a32a-edc04ce61c09","Type":"ContainerStarted","Data":"e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668"}
Nov 02 17:32:28.858587 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.858550    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerStarted","Data":"83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017"}
Nov 02 17:32:28.868012 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.867979    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a"}
Nov 02 17:32:28.882009 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.881966    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m9pzk" event={"ID":"64d7830f-395c-4694-a61c-c25f1c0adc5a","Type":"ContainerStarted","Data":"24c880010b914afc7edcce914d182aca9278f2d41eda292938760debd5a83897"}
Nov 02 17:32:28.893853 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:28.893800    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dv585" event={"ID":"7e1fc379-cd57-4b14-be52-b3975a5f6514","Type":"ContainerStarted","Data":"17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0"}
Nov 02 17:32:29.017785 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.017712    2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:32:29.415257 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.415171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:29.415468 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.415270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:29.415468 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.415397    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:29.415468 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.415460    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret podName:5188d247-bf34-437c-bde7-0abc73c312f5 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:31.415443599 +0000 UTC m=+5.142065394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret") pod "global-pull-secret-syncer-b26bx" (UID: "5188d247-bf34-437c-bde7-0abc73c312f5") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:29.415905 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.415871    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:29.415990 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.415922    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:31.41590866 +0000 UTC m=+5.142530454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:29.516715 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.516671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:29.516920 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.516867    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:32:29.516920 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.516889    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:32:29.516920 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.516901    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hwsfg for pod openshift-network-diagnostics/network-check-target-pp7tz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:29.517093 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.516962    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg podName:812e2a84-04de-4c0e-b8f1-f956381cb3fd nodeName:}" failed. No retries permitted until 2026-11-02 17:32:31.516943607 +0000 UTC m=+5.243565392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwsfg" (UniqueName: "kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg") pod "network-check-target-pp7tz" (UID: "812e2a84-04de-4c0e-b8f1-f956381cb3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:29.737493 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.737379    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:27 +0000 UTC" deadline="2028-04-28 03:14:30.975544619 +0000 UTC"
Nov 02 17:32:29.737493 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.737438    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13017h42m1.238109921s"
Nov 02 17:32:29.763167 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.763125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:29.763354 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.763273    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:29.763698 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:29.763678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:29.763818 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:29.763797    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:30.306562 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:30.306503    2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:32:30.764072 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:30.764041    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:30.764520 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:30.764183    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:31.434891 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.434857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:31.435086 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.434916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:31.435145 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.435091    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:31.435196 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.435151    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:35.435130522 +0000 UTC m=+9.161752302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:31.435516 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.435479    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:31.435631 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.435526    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret podName:5188d247-bf34-437c-bde7-0abc73c312f5 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:35.435514425 +0000 UTC m=+9.162136205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret") pod "global-pull-secret-syncer-b26bx" (UID: "5188d247-bf34-437c-bde7-0abc73c312f5") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:31.536006 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.535971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:31.536160 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.536145    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:32:31.536217 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.536171    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:32:31.536217 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.536185    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hwsfg for pod openshift-network-diagnostics/network-check-target-pp7tz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:31.536317 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.536240    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg podName:812e2a84-04de-4c0e-b8f1-f956381cb3fd nodeName:}" failed. No retries permitted until 2026-11-02 17:32:35.536221333 +0000 UTC m=+9.262843127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwsfg" (UniqueName: "kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg") pod "network-check-target-pp7tz" (UID: "812e2a84-04de-4c0e-b8f1-f956381cb3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:31.764012 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.763741    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:31.764179 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.764127    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:31.764618 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.764598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:31.764728 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:31.764705    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:31.918082 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.918043    2566 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="c8859f2dab6a8f85135522dedfaadd16cc8532abeeeb09b05dfb1643cae95c52" exitCode=0
Nov 02 17:32:31.918263 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:31.918093    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerDied","Data":"c8859f2dab6a8f85135522dedfaadd16cc8532abeeeb09b05dfb1643cae95c52"}
Nov 02 17:32:32.766781 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:32.766271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:32.766781 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:32.766417    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:32.944207 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:32.944157    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fh7z4" event={"ID":"d806b75d-88ca-4e51-94c3-cd45fd8888db","Type":"ContainerStarted","Data":"a96ccbf999fe7b5df3e03befb87e1b97a1ea3e8ed7ccf6bd55211d35343faf74"}
Nov 02 17:32:32.951904 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:32.951819    2566 generic.go:356] "Generic (PLEG): container finished" podID="22d0975639c634403c0017e572800bfa" containerID="d817de27ce2c3085a921c1cfcd4f736a48dbdcaa7438bf66962375871300c76e" exitCode=0
Nov 02 17:32:32.951904 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:32.951894    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"22d0975639c634403c0017e572800bfa","Type":"ContainerDied","Data":"d817de27ce2c3085a921c1cfcd4f736a48dbdcaa7438bf66962375871300c76e"}
Nov 02 17:32:32.981442 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:32.981378    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fh7z4" podStartSLOduration=2.65875838 podStartE2EDuration="6.981347705s" totalImagesPullingTime="4.322589325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:32.964231965 +0000 UTC m=+6.690853769" watchObservedRunningTime="2026-11-02 17:32:32.981347705 +0000 UTC m=+6.707969508"
Nov 02 17:32:33.763310 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:33.763272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:33.763486 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:33.763419    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:33.763855 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:33.763817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:33.763962 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:33.763939    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:33.959068 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:33.959036    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"4cabb9f1887dd6d87ae5b4fba788119d","Type":"ContainerStarted","Data":"eb6514e0689b546289175346c7c885fe5db651ba3faf64052fda2dabc20df2ac"}
Nov 02 17:32:33.963783 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:33.963750    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"22d0975639c634403c0017e572800bfa","Type":"ContainerStarted","Data":"4f21cbfd88444e763d335be4cc60157b05b725f4ed582b3da1edc6cf096af565"}
Nov 02 17:32:33.994623 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:33.994563    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" podStartSLOduration=6.994546101 podStartE2EDuration="6.994546101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:33.993647469 +0000 UTC m=+7.720269284" watchObservedRunningTime="2026-11-02 17:32:33.994546101 +0000 UTC m=+7.721167904"
Nov 02 17:32:33.994815 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:33.994673    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal" podStartSLOduration=6.994666055 podStartE2EDuration="6.994666055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:33.977552184 +0000 UTC m=+7.704173989" watchObservedRunningTime="2026-11-02 17:32:33.994666055 +0000 UTC m=+7.721287858"
Nov 02 17:32:34.765862 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:34.765775    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:34.766033 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:34.765914    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:34.968733 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:34.968702    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dv585" event={"ID":"7e1fc379-cd57-4b14-be52-b3975a5f6514","Type":"ContainerStarted","Data":"e8fea402c4e57c53cbcfbe37626f6af2d3539a75244d578aff9bde9ca518111e"}
Nov 02 17:32:34.974139 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:34.972878    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2" event={"ID":"b90ebafb-a030-43ce-a32a-edc04ce61c09","Type":"ContainerStarted","Data":"5689223958c9fb303eea684e8ca8641025de18c576091fabdddcce98e38c8aad"}
Nov 02 17:32:35.470191 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.470100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:35.470332 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.470214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:35.470332 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.470325    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:35.470421 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.470379    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret podName:5188d247-bf34-437c-bde7-0abc73c312f5 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:43.470366488 +0000 UTC m=+17.196988281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret") pod "global-pull-secret-syncer-b26bx" (UID: "5188d247-bf34-437c-bde7-0abc73c312f5") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:35.470827 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.470807    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:35.470922 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.470872    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:43.47085987 +0000 UTC m=+17.197481662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:35.571135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.571091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:35.571323 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.571266    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:32:35.571323 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.571287    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:32:35.571323 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.571303    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hwsfg for pod openshift-network-diagnostics/network-check-target-pp7tz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:35.571477 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.571364    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg podName:812e2a84-04de-4c0e-b8f1-f956381cb3fd nodeName:}" failed. No retries permitted until 2026-11-02 17:32:43.571344763 +0000 UTC m=+17.297966562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwsfg" (UniqueName: "kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg") pod "network-check-target-pp7tz" (UID: "812e2a84-04de-4c0e-b8f1-f956381cb3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:35.763418 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.763344    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:35.763568 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.763475    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:35.763637 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.763615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:35.763764 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:35.763702    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:35.972204 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.972178    2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:32:35.982314 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:35.982241    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2" event={"ID":"b90ebafb-a030-43ce-a32a-edc04ce61c09","Type":"ContainerStarted","Data":"dcfae1ce9b685d865b8b95a8339f7c58b125a25c44fd01bd13c7aa7378af69bd"}
Nov 02 17:32:36.764586 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.764118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:36.764586 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:36.764253    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:36.766953 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.766369    2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:32:35.972200337Z","UUID":"a1cf311d-2256-4e3d-aca7-ab96e154e908","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:32:36.770133 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.770110    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:32:36.770133 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.770135    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:32:36.988067 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.987753    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7lh5w" event={"ID":"505743f8-9737-44a8-987b-62f5d35212d3","Type":"ContainerStarted","Data":"d84406960138bdb0ffb0dcebee262a599658ba85e6ca683b8ab2f5ccb9e9aa46"}
Nov 02 17:32:36.993961 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.993928    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerStarted","Data":"b01f8bc1e20f42d679f3e51396e5f77850c7d6fa0749118d3c5ad67ee93733da"}
Nov 02 17:32:36.997431 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:36.997165    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzlmp" event={"ID":"032f9f33-d723-46ad-9d89-45956596eec5","Type":"ContainerStarted","Data":"0a3b1197511466ef57cd98cb732bf6acf0069d111f04b58948854f11f2de451f"}
Nov 02 17:32:37.007389 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.007328    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dv585" podStartSLOduration=4.224686189 podStartE2EDuration="10.00731066s" totalImagesPullingTime="5.782624471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:34.98828304 +0000 UTC m=+8.714904841" watchObservedRunningTime="2026-11-02 17:32:37.00731066 +0000 UTC m=+10.733932464"
Nov 02 17:32:37.028498 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.028434    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7lh5w" podStartSLOduration=2.929267838 podStartE2EDuration="11.02841368s" totalImagesPullingTime="8.099145842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:37.007886314 +0000 UTC m=+10.734508121" watchObservedRunningTime="2026-11-02 17:32:37.02841368 +0000 UTC m=+10.755035482"
Nov 02 17:32:37.251397 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.250856    2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:37.251626 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.251604    2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:37.268821 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.268545    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dzlmp" podStartSLOduration=1.62073693 podStartE2EDuration="10.268524111s" totalImagesPullingTime="8.647787181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:37.043888126 +0000 UTC m=+10.770509927" watchObservedRunningTime="2026-11-02 17:32:37.268524111 +0000 UTC m=+10.995145917"
Nov 02 17:32:37.763695 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.763658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:37.763873 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:37.763770    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:37.763873 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:37.763783    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:37.763977 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:37.763878    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:38.002277 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:38.002193    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2" event={"ID":"b90ebafb-a030-43ce-a32a-edc04ce61c09","Type":"ContainerStarted","Data":"c1023117635589cd59e00814dda5732e8ddfb5c30faa6ca7f3b1ed4239d525d5"}
Nov 02 17:32:38.004551 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:38.004510    2566 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="b01f8bc1e20f42d679f3e51396e5f77850c7d6fa0749118d3c5ad67ee93733da" exitCode=0
Nov 02 17:32:38.004691 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:38.004605    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerDied","Data":"b01f8bc1e20f42d679f3e51396e5f77850c7d6fa0749118d3c5ad67ee93733da"}
Nov 02 17:32:38.026221 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:38.026166    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2" podStartSLOduration=3.277421501 podStartE2EDuration="12.026151808s" totalImagesPullingTime="8.748730307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:38.02469757 +0000 UTC m=+11.751319372" watchObservedRunningTime="2026-11-02 17:32:38.026151808 +0000 UTC m=+11.752773610"
Nov 02 17:32:38.763924 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:38.763896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:38.764057 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:38.764038    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:39.009479 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.009400    2566 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="524e9066f6637451dac0e90f0253f97d74ab49243a83f0aba60480c3a82de32b" exitCode=0
Nov 02 17:32:39.010147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.009481    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerDied","Data":"524e9066f6637451dac0e90f0253f97d74ab49243a83f0aba60480c3a82de32b"}
Nov 02 17:32:39.011805 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.011776    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m9pzk" event={"ID":"64d7830f-395c-4694-a61c-c25f1c0adc5a","Type":"ContainerStarted","Data":"6dfaae2c9572c1008aea92424dc18cba665fd395c9bcc50d1b13ded3489913dd"}
Nov 02 17:32:39.013728 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.013699    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-msvsr" event={"ID":"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce","Type":"ContainerStarted","Data":"ee7b28c272601d498fb638d3db36a49ec207651a613c0027962ec3fc9eaabafa"}
Nov 02 17:32:39.047465 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.047406    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-msvsr" podStartSLOduration=4.285070154 podStartE2EDuration="13.047388848s" totalImagesPullingTime="8.762318694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:39.047158711 +0000 UTC m=+12.773780513" watchObservedRunningTime="2026-11-02 17:32:39.047388848 +0000 UTC m=+12.774010651"
Nov 02 17:32:39.068409 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.068358    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m9pzk" podStartSLOduration=1.509739028 podStartE2EDuration="12.068344551s" totalImagesPullingTime="10.558605523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:39.067580585 +0000 UTC m=+12.794202414" watchObservedRunningTime="2026-11-02 17:32:39.068344551 +0000 UTC m=+12.794966352"
Nov 02 17:32:39.763334 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.763298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:39.763484 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:39.763421    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:39.763558 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:39.763478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:39.763630 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:39.763608    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:40.763448 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:40.763407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:40.764137 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:40.763570    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:41.022001 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:41.021902    2566 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="da4a681bf350a399c37498d4164e12877a476bce5a0c77acab916c6a0cefb5dc" exitCode=0
Nov 02 17:32:41.022001 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:41.021952    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerDied","Data":"da4a681bf350a399c37498d4164e12877a476bce5a0c77acab916c6a0cefb5dc"}
Nov 02 17:32:41.763700 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:41.763665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:41.763700 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:41.763706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:41.764274 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:41.763791    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:41.764274 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:41.763959    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:42.763875 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:42.763681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:42.764320 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:42.763995    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:43.536928 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:43.536679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:43.537112 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:43.536945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:43.537112 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.536823    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:43.537112 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.537060    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret podName:5188d247-bf34-437c-bde7-0abc73c312f5 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.537036922 +0000 UTC m=+33.263658727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret") pod "global-pull-secret-syncer-b26bx" (UID: "5188d247-bf34-437c-bde7-0abc73c312f5") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:32:43.537112 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.537094    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:43.537265 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.537147    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.537132276 +0000 UTC m=+33.263754074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:32:43.637774 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:43.637700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:43.637920 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.637866    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:32:43.637920 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.637884    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:32:43.637920 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.637894    2566 projected.go:196] Error preparing data for projected volume kube-api-access-hwsfg for pod openshift-network-diagnostics/network-check-target-pp7tz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:43.638040 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.637944    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg podName:812e2a84-04de-4c0e-b8f1-f956381cb3fd nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.637930041 +0000 UTC m=+33.364551834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hwsfg" (UniqueName: "kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg") pod "network-check-target-pp7tz" (UID: "812e2a84-04de-4c0e-b8f1-f956381cb3fd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:32:43.763927 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:43.763895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:43.763927 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:43.763931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:43.764436 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.764009    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:43.764436 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:43.764163    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:44.032231 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.032133    2566 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="3420fd0e9762d0f1537de79905d1f6726ba3d257d8dfaec76ef2984d54a783e8" exitCode=0
Nov 02 17:32:44.032431 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.032221    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerDied","Data":"3420fd0e9762d0f1537de79905d1f6726ba3d257d8dfaec76ef2984d54a783e8"}
Nov 02 17:32:44.039440 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.039341    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:32:44.041497 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041448    2566 generic.go:356] "Generic (PLEG): container finished" podID="269ca06c-b59b-496f-81aa-a08c525d0098" containerID="73fe9091f868c36d1fc896ecbd71b0f754b58e012d934a895c98ae03ccbef32d" exitCode=1
Nov 02 17:32:44.041497 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041478    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"986afaaa4e52b2f9c3fc2f9f2b6dcc6f0bf13dfe5227230d18568a486bd31d88"}
Nov 02 17:32:44.041607 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041593    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"27fb0d78258b7046a311a0d63b56e813694ae829863f83df4d2dbe11a17d1ac7"}
Nov 02 17:32:44.041660 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041644    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"6d7bdf119b7a0518f4a379a10bb67be4f7d8fcaec8e4a0589d337e2f37e12353"}
Nov 02 17:32:44.041716 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041693    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"acd012a303386067344ffc75aaac84b998a0173008142cd6ff9ea121b1cb2dd8"}
Nov 02 17:32:44.041768 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041741    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerDied","Data":"73fe9091f868c36d1fc896ecbd71b0f754b58e012d934a895c98ae03ccbef32d"}
Nov 02 17:32:44.041821 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.041788    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"b49635e431de26f95ff6c30a0ae8719df961dbebe5eae1db5ed517d3b0dcd9f9"}
Nov 02 17:32:44.763089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:44.763058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:44.763259 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:44.763167    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:45.046440 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.046354    2566 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="a28331119236f050979c3695b9343f1075ce17d99e6815da5d31d17ba3477805" exitCode=0
Nov 02 17:32:45.046440 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.046394    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerDied","Data":"a28331119236f050979c3695b9343f1075ce17d99e6815da5d31d17ba3477805"}
Nov 02 17:32:45.069053 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.069018    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:45.069185 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.069162    2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:32:45.069728 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.069703    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:32:45.763572 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.763526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:45.763719 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:45.763526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:45.763719 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:45.763655    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:45.763719 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:45.763712    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:46.051492 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:46.051451    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vng2x" event={"ID":"c58d6490-3f26-48cc-95eb-440097888b15","Type":"ContainerStarted","Data":"1db8fe248503201a2f96de3db5abe3550addb19ea3d6be4121b92cb0484652bd"}
Nov 02 17:32:46.057494 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:46.057462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:32:46.057885 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:46.057864    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"2ec354a0983986f96c46f5edb698a3bea93a88a7afe360bc4d5d199e10574c95"}
Nov 02 17:32:46.074379 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:46.074336    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vng2x" podStartSLOduration=8.876451288 podStartE2EDuration="20.074323582s" totalImagesPullingTime="11.197872294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:46.07287846 +0000 UTC m=+19.799500261" watchObservedRunningTime="2026-11-02 17:32:46.074323582 +0000 UTC m=+19.800945384"
Nov 02 17:32:46.763731 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:46.763691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:46.763935 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:46.763793    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:47.763337 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:47.763306    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:47.763337 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:47.763331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:47.763965 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:47.763421    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:47.763965 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:47.763554    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:48.065298 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.065133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:32:48.065658 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.065635    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"909c8aaf1e6696f570933b221e209018b3a6e093bfbb72c343da16f62ed69ee9"}
Nov 02 17:32:48.065900 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.065875    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:48.066013 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.065952    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:48.066013 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.065996    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:48.066106 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.066049    2566 scope.go:122] "RemoveContainer" containerID="73fe9091f868c36d1fc896ecbd71b0f754b58e012d934a895c98ae03ccbef32d"
Nov 02 17:32:48.083107 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.083087    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:48.083193 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.083187    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:32:48.763152 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:48.763118    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:48.763295 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:48.763255    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:49.072330 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.072262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:32:49.072761 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.072605    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" event={"ID":"269ca06c-b59b-496f-81aa-a08c525d0098","Type":"ContainerStarted","Data":"e67018706b366516d076ae3604452d54bc80ffdf0b4351dbb32b2ef27b58b0c1"}
Nov 02 17:32:49.101897 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.101831    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg" podStartSLOduration=7.330037451 podStartE2EDuration="22.101817648s" totalImagesPullingTime="14.771780197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:49.100009017 +0000 UTC m=+22.826630819" watchObservedRunningTime="2026-11-02 17:32:49.101817648 +0000 UTC m=+22.828439450"
Nov 02 17:32:49.367359 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.367270    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b26bx"]
Nov 02 17:32:49.367508 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.367456    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:49.367654 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:49.367588    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:49.367979 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.367949    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pp7tz"]
Nov 02 17:32:49.368072 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.368059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:49.368222 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:49.368199    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:49.369634 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.369612    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bq652"]
Nov 02 17:32:49.369724 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:49.369703    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:49.369803 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:49.369784    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:50.763187 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:50.763151    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:50.763679 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:50.763157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:50.763679 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:50.763265    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:50.763679 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:50.763334    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:51.763905 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:51.763875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:51.764321 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:51.763983    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:52.763354 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:52.763318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:52.763538 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:52.763499    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:52.763607 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:52.763546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:52.763665 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:52.763648    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:53.763673 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:53.763638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:53.764027 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:53.763743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:54.763489 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:54.763460    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:54.763659 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:54.763463    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:54.763659 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:54.763560    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pp7tz" podUID="812e2a84-04de-4c0e-b8f1-f956381cb3fd"
Nov 02 17:32:54.764001 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:54.763667    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bq652" podUID="fd30ffd8-a9b2-46a3-b057-e25e98540942"
Nov 02 17:32:55.763626 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:55.763445    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:55.763746 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:55.763717    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b26bx" podUID="5188d247-bf34-437c-bde7-0abc73c312f5"
Nov 02 17:32:56.070082 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.070056    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeReady"
Nov 02 17:32:56.070262 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.070185    2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 17:32:56.103911 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.103877    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nwfc8"]
Nov 02 17:32:56.107257 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.107200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:32:56.110559 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.108786    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-892gj"]
Nov 02 17:32:56.114282 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.114149    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-622zn"]
Nov 02 17:32:56.114657 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.114257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:32:56.115537 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.115514    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.115701 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.115678    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.118361 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.118342    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bxzmw"]
Nov 02 17:32:56.118469 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.118439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.121458 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.121434    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fd6588566-jdf7q"]
Nov 02 17:32:56.121922 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.121901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:32:56.124621 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.124600    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"]
Nov 02 17:32:56.124748 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.124730    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.129418 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.129385    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 17:32:56.129581 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.129558    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2cgvl\""
Nov 02 17:32:56.130600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.130285    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"]
Nov 02 17:32:56.132444 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.132422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:32:56.132953 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.132929    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 17:32:56.132953 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.132940    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 17:32:56.133158 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.133140    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 17:32:56.133234 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.133215    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpng6\""
Nov 02 17:32:56.135165 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.133447    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"]
Nov 02 17:32:56.135434 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.135401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.140003 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.139981    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-hk6xf"]
Nov 02 17:32:56.140420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.140177    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.143981 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.143955    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-47kbn"]
Nov 02 17:32:56.144366 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.144346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.152515 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.152487    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"]
Nov 02 17:32:56.152817 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.152791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.164867 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.161046    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7987d8c6c-fplrw"]
Nov 02 17:32:56.164867 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.161181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.165146 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.165060    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c7d5754d7-v6znk"]
Nov 02 17:32:56.165198 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.165180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.168969 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.168948    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"]
Nov 02 17:32:56.169089 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.169078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.171922 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.171902    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"]
Nov 02 17:32:56.172042 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.172025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.174633 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.174618    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nwfc8"]
Nov 02 17:32:56.174722 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.174643    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gfg92"]
Nov 02 17:32:56.174722 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.174714    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:32:56.178187 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.178170    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-892gj"]
Nov 02 17:32:56.178260 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.178200    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gqgqg"]
Nov 02 17:32:56.178313 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.178282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.181015 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.180998    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"]
Nov 02 17:32:56.181087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181022    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bxzmw"]
Nov 02 17:32:56.181087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181035    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"]
Nov 02 17:32:56.181087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181049    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"]
Nov 02 17:32:56.181087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181067    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd6588566-jdf7q"]
Nov 02 17:32:56.181087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181079    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-47kbn"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181093    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7987d8c6c-fplrw"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181105    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c7d5754d7-v6znk"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181118    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gfg92"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181133    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-622zn"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181149    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181162    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gqgqg"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181176    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181197    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"]
Nov 02 17:32:56.181268 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.181214    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-hk6xf"]
Nov 02 17:32:56.215883 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.215860    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 17:32:56.216132 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216114    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216205 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216120    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 17:32:56.216205 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216122    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216205 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216139    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 17:32:56.216205 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216201    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.216393 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216254    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:32:56.216393 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216255    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.216393 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216274    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 17:32:56.216393 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216306    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216393 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216293    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.216393 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216394    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216591 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216511    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 17:32:56.216591 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216521    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.216591 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216548    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216700 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216644    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216700 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216662    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.216700 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216675    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.216886 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216867    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:32:56.216886 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216878    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.216886 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.216883    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.217490 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.217472    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.217490 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.217484    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.217868 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.217832    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.217868 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.217832    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.217996 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.217890    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:32:56.217996 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.217900    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:32:56.218143 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.218127    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 17:32:56.223873 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.223832    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 17:32:56.229990 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.229961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq28j\" (UniqueName: \"kubernetes.io/projected/17ccbff4-2818-4e70-b2f4-a64c92f789e8-kube-api-access-nq28j\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.230101 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52a1ad24-d5c4-49cf-aa71-b92f14918879-serving-cert\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.230101 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwm2f\" (UniqueName: \"kubernetes.io/projected/52a1ad24-d5c4-49cf-aa71-b92f14918879-kube-api-access-fwm2f\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.230101 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.230101 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.230297 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.230297 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b497b468-2062-4cc3-9687-0994f99979fd-config\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.230389 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.230389 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17ccbff4-2818-4e70-b2f4-a64c92f789e8-serving-cert\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.230477 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.230477 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.230477 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.230600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.230600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.230600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.230600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.230600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqcnt\" (UniqueName: \"kubernetes.io/projected/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-kube-api-access-xqcnt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.230600 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.230866 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.230866 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.230866 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-service-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.230866 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.230866 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.230794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vls44\" (UniqueName: \"kubernetes.io/projected/565006a5-a1dd-4383-8078-f99d66892a75-kube-api-access-vls44\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.231135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.231187 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.231263 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-snapshots\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.231345 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.231409 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.231457 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.231510 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.231580 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99htf\" (UniqueName: \"kubernetes.io/projected/d55feb08-7024-4795-9a42-7345567fe78b-kube-api-access-99htf\") pod \"volume-data-source-validator-58c9f5f46d-4559l\" (UID: \"d55feb08-7024-4795-9a42-7345567fe78b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:32:56.231627 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b497b468-2062-4cc3-9687-0994f99979fd-serving-cert\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.231680 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.231772 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bczck\" (UniqueName: \"kubernetes.io/projected/814ca1bd-35f4-4b88-9d95-f9e1cc47845b-kube-api-access-bczck\") pod \"kas-connection-checker-685895c7c9-892gj\" (UID: \"814ca1bd-35f4-4b88-9d95-f9e1cc47845b\") " pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:32:56.231815 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-stats-auth\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.231900 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4x8\" (UniqueName: \"kubernetes.io/projected/b497b468-2062-4cc3-9687-0994f99979fd-kube-api-access-6j4x8\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.232017 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231931    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 17:32:56.232017 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.231921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2n6\" (UniqueName: \"kubernetes.io/projected/d4e48a54-0431-4e67-a886-30ce38fc01ef-kube-api-access-6m2n6\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.232114 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.232025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.233227 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjlz\" (UniqueName: \"kubernetes.io/projected/f3c68c66-cc5c-4bc4-9937-691ece1b51b3-kube-api-access-hxjlz\") pod \"kas-connection-checker-685895c7c9-nwfc8\" (UID: \"f3c68c66-cc5c-4bc4-9937-691ece1b51b3\") " pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:32:56.233346 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233248    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.233410 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233362    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-default-certificate\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.233630 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.233711 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dq8\" (UniqueName: \"kubernetes.io/projected/7adb34aa-6a54-4a58-8761-49b5de183bba-kube-api-access-p8dq8\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.233769 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233753    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z5gr8\""
Nov 02 17:32:56.233889 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.233825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.234060 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234044    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9bqkr\""
Nov 02 17:32:56.234111 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.234220 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-tmp\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.234292 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234278    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xgjc4\""
Nov 02 17:32:56.234398 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234384    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 17:32:56.234463 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234450    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 17:32:56.234668 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234653    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-v5pqt\""
Nov 02 17:32:56.235017 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235001    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nq4sl\""
Nov 02 17:32:56.235100 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.234278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/565006a5-a1dd-4383-8078-f99d66892a75-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.235100 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235060    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 17:32:56.235100 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-config\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.235100 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235095    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l5rdl\""
Nov 02 17:32:56.235298 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xszp\" (UniqueName: \"kubernetes.io/projected/a893c12e-173e-4d9c-8080-8fa9992d9396-kube-api-access-6xszp\") pod \"kas-connection-checker-685895c7c9-bxzmw\" (UID: \"a893c12e-173e-4d9c-8080-8fa9992d9396\") " pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:32:56.235298 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235272    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 17:32:56.235298 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235276    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 17:32:56.235430 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235412    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 17:32:56.235471 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235429    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:32:56.235520 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235472    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s9lzk\""
Nov 02 17:32:56.235561 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235545    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 17:32:56.235606 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235564    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:32:56.235780 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.235756    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-76wwc\""
Nov 02 17:32:56.237143 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.237125    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 17:32:56.237348 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.237321    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mfww7\""
Nov 02 17:32:56.237432 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.237371    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9wn4g\""
Nov 02 17:32:56.237432 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.237377    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 17:32:56.237535 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.237525    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 17:32:56.237581 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.237542    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dp4wq\""
Nov 02 17:32:56.238249 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.238235    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:32:56.336030 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.335940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.336030 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.335987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.336030 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.336280 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.336280 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99htf\" (UniqueName: \"kubernetes.io/projected/d55feb08-7024-4795-9a42-7345567fe78b-kube-api-access-99htf\") pod \"volume-data-source-validator-58c9f5f46d-4559l\" (UID: \"d55feb08-7024-4795-9a42-7345567fe78b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:32:56.336280 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b497b468-2062-4cc3-9687-0994f99979fd-serving-cert\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.336378 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.336378 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczck\" (UniqueName: \"kubernetes.io/projected/814ca1bd-35f4-4b88-9d95-f9e1cc47845b-kube-api-access-bczck\") pod \"kas-connection-checker-685895c7c9-892gj\" (UID: \"814ca1bd-35f4-4b88-9d95-f9e1cc47845b\") " pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:32:56.336378 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-stats-auth\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.336378 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4x8\" (UniqueName: \"kubernetes.io/projected/b497b468-2062-4cc3-9687-0994f99979fd-kube-api-access-6j4x8\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.336509 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2n6\" (UniqueName: \"kubernetes.io/projected/d4e48a54-0431-4e67-a886-30ce38fc01ef-kube-api-access-6m2n6\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.336509 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.336509 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336428    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.336509 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjlz\" (UniqueName: \"kubernetes.io/projected/f3c68c66-cc5c-4bc4-9937-691ece1b51b3-kube-api-access-hxjlz\") pod \"kas-connection-checker-685895c7c9-nwfc8\" (UID: \"f3c68c66-cc5c-4bc4-9937-691ece1b51b3\") " pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:32:56.336509 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.336669 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-default-certificate\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.336669 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.336669 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dq8\" (UniqueName: \"kubernetes.io/projected/7adb34aa-6a54-4a58-8761-49b5de183bba-kube-api-access-p8dq8\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.336669 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63048823-3df5-48fa-a65d-c1d789a319c9-tmp-dir\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.336669 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.336669 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336654    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f693f58-c274-4b42-863d-7b4e73d8fb10-cert\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.336878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.336878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqh9\" (UniqueName: \"kubernetes.io/projected/1f693f58-c274-4b42-863d-7b4e73d8fb10-kube-api-access-xjqh9\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.336878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-tmp\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.336878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/565006a5-a1dd-4383-8078-f99d66892a75-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.336878 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63048823-3df5-48fa-a65d-c1d789a319c9-config-volume\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.337038 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-config\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.337038 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xszp\" (UniqueName: \"kubernetes.io/projected/a893c12e-173e-4d9c-8080-8fa9992d9396-kube-api-access-6xszp\") pod \"kas-connection-checker-685895c7c9-bxzmw\" (UID: \"a893c12e-173e-4d9c-8080-8fa9992d9396\") " pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:32:56.337038 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336941    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq28j\" (UniqueName: \"kubernetes.io/projected/17ccbff4-2818-4e70-b2f4-a64c92f789e8-kube-api-access-nq28j\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.337038 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.336976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52a1ad24-d5c4-49cf-aa71-b92f14918879-serving-cert\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.337038 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwm2f\" (UniqueName: \"kubernetes.io/projected/52a1ad24-d5c4-49cf-aa71-b92f14918879-kube-api-access-fwm2f\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.337038 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337034    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b497b468-2062-4cc3-9687-0994f99979fd-config\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17ccbff4-2818-4e70-b2f4-a64c92f789e8-serving-cert\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.337302 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcnt\" (UniqueName: \"kubernetes.io/projected/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-kube-api-access-xqcnt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337528    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-tmp\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.337638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-service-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vls44\" (UniqueName: \"kubernetes.io/projected/565006a5-a1dd-4383-8078-f99d66892a75-kube-api-access-vls44\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8vv\" (UniqueName: \"kubernetes.io/projected/63048823-3df5-48fa-a65d-c1d789a319c9-kube-api-access-wc8vv\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcm2m\" (UniqueName: \"kubernetes.io/projected/be9f38d9-38b2-42e3-8236-a3fa68f311db-kube-api-access-jcm2m\") pod \"network-check-source-6f78d59f99-bbsgc\" (UID: \"be9f38d9-38b2-42e3-8236-a3fa68f311db\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.338147 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.337954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-snapshots\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.338527 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.338273    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/565006a5-a1dd-4383-8078-f99d66892a75-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.338527 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.338489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-snapshots\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.339143 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.338854    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-config\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.339143 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.339082    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b497b468-2062-4cc3-9687-0994f99979fd-config\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.339312 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.339174    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:32:56.339312 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.339268    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.839247594 +0000 UTC m=+30.565869388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:32:56.340069 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.340050    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:56.341452 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.341418    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.841398585 +0000 UTC m=+30.568020365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:56.341452 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.340863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.341622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.340884    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.341622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.341322    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.341622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.340566    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.341622 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.340269    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:32:56.341622 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.341543    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.841531052 +0000 UTC m=+30.568152836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:32:56.341891 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.341666    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:56.341891 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.341678    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:32:56.341891 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.341715    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.841701856 +0000 UTC m=+30.568323639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:32:56.342036 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.342017    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:56.342036 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.342033    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:32:56.342125 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.342075    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.842059886 +0000 UTC m=+30.568681685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:32:56.342584 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.342564    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.342817 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.342792    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.343214 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.343188    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.343387 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.343366    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.843347532 +0000 UTC m=+30.569969514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : configmap references non-existent config key: service-ca.crt
Nov 02 17:32:56.343504 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.343478    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.343663 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.343640    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b497b468-2062-4cc3-9687-0994f99979fd-serving-cert\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.343751 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.343654    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-service-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.343951 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.343933    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:32:56.344028 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.344020    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.844005841 +0000 UTC m=+30.570627628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:32:56.344094 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.343972    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.344622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.344535    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.344622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.344578    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.344622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.344591    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-stats-auth\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.344622 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.344616    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.345170 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.345151    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99htf\" (UniqueName: \"kubernetes.io/projected/d55feb08-7024-4795-9a42-7345567fe78b-kube-api-access-99htf\") pod \"volume-data-source-validator-58c9f5f46d-4559l\" (UID: \"d55feb08-7024-4795-9a42-7345567fe78b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:32:56.345285 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.345264    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.345350 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.345322    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.346470 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.346442    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczck\" (UniqueName: \"kubernetes.io/projected/814ca1bd-35f4-4b88-9d95-f9e1cc47845b-kube-api-access-bczck\") pod \"kas-connection-checker-685895c7c9-892gj\" (UID: \"814ca1bd-35f4-4b88-9d95-f9e1cc47845b\") " pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:32:56.346566 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.346505    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-default-certificate\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.346787 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.346763    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17ccbff4-2818-4e70-b2f4-a64c92f789e8-serving-cert\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.346942 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.346921    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52a1ad24-d5c4-49cf-aa71-b92f14918879-serving-cert\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.347922 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.347903    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.348544 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.348516    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xszp\" (UniqueName: \"kubernetes.io/projected/a893c12e-173e-4d9c-8080-8fa9992d9396-kube-api-access-6xszp\") pod \"kas-connection-checker-685895c7c9-bxzmw\" (UID: \"a893c12e-173e-4d9c-8080-8fa9992d9396\") " pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:32:56.350062 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.350036    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.351218 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.351189    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcnt\" (UniqueName: \"kubernetes.io/projected/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-kube-api-access-xqcnt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.351299 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.351214    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2n6\" (UniqueName: \"kubernetes.io/projected/d4e48a54-0431-4e67-a886-30ce38fc01ef-kube-api-access-6m2n6\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.351604 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.351579    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjlz\" (UniqueName: \"kubernetes.io/projected/f3c68c66-cc5c-4bc4-9937-691ece1b51b3-kube-api-access-hxjlz\") pod \"kas-connection-checker-685895c7c9-nwfc8\" (UID: \"f3c68c66-cc5c-4bc4-9937-691ece1b51b3\") " pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:32:56.352344 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.352319    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.352442 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.352409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4x8\" (UniqueName: \"kubernetes.io/projected/b497b468-2062-4cc3-9687-0994f99979fd-kube-api-access-6j4x8\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.352442 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.352420    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq28j\" (UniqueName: \"kubernetes.io/projected/17ccbff4-2818-4e70-b2f4-a64c92f789e8-kube-api-access-nq28j\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.353087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.353051    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.355616 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.355586    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dq8\" (UniqueName: \"kubernetes.io/projected/7adb34aa-6a54-4a58-8761-49b5de183bba-kube-api-access-p8dq8\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.355928 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.355910    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vls44\" (UniqueName: \"kubernetes.io/projected/565006a5-a1dd-4383-8078-f99d66892a75-kube-api-access-vls44\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.356634 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.356611    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwm2f\" (UniqueName: \"kubernetes.io/projected/52a1ad24-d5c4-49cf-aa71-b92f14918879-kube-api-access-fwm2f\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.428713 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.428681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:32:56.438518 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8vv\" (UniqueName: \"kubernetes.io/projected/63048823-3df5-48fa-a65d-c1d789a319c9-kube-api-access-wc8vv\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.438518 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm2m\" (UniqueName: \"kubernetes.io/projected/be9f38d9-38b2-42e3-8236-a3fa68f311db-kube-api-access-jcm2m\") pod \"network-check-source-6f78d59f99-bbsgc\" (UID: \"be9f38d9-38b2-42e3-8236-a3fa68f311db\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:32:56.438790 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.438790 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63048823-3df5-48fa-a65d-c1d789a319c9-tmp-dir\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.438790 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f693f58-c274-4b42-863d-7b4e73d8fb10-cert\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.438790 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqh9\" (UniqueName: \"kubernetes.io/projected/1f693f58-c274-4b42-863d-7b4e73d8fb10-kube-api-access-xjqh9\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.438790 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438680    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63048823-3df5-48fa-a65d-c1d789a319c9-config-volume\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.438790 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.438764    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:32:56.439086 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.438896    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:56.93887779 +0000 UTC m=+30.665499588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:32:56.439086 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.438962    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63048823-3df5-48fa-a65d-c1d789a319c9-tmp-dir\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.439283 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.439265    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63048823-3df5-48fa-a65d-c1d789a319c9-config-volume\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.441431 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.441357    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f693f58-c274-4b42-863d-7b4e73d8fb10-cert\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.441431 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.441436    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:32:56.450933 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.450911    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8vv\" (UniqueName: \"kubernetes.io/projected/63048823-3df5-48fa-a65d-c1d789a319c9-kube-api-access-wc8vv\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.451014 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.450935    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqh9\" (UniqueName: \"kubernetes.io/projected/1f693f58-c274-4b42-863d-7b4e73d8fb10-kube-api-access-xjqh9\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.451060 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.451018    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcm2m\" (UniqueName: \"kubernetes.io/projected/be9f38d9-38b2-42e3-8236-a3fa68f311db-kube-api-access-jcm2m\") pod \"network-check-source-6f78d59f99-bbsgc\" (UID: \"be9f38d9-38b2-42e3-8236-a3fa68f311db\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:32:56.465023 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.464992    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:32:56.470882 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.470695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:32:56.477798 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.477773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:32:56.483831 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.483696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:32:56.496374 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.496198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:32:56.541815 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.540096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:32:56.541815 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.540875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:32:56.549594 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.549520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:32:56.731519 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.731434    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-892gj"]
Nov 02 17:32:56.734254 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.734228    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nwfc8"]
Nov 02 17:32:56.734974 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.734945    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814ca1bd_35f4_4b88_9d95_f9e1cc47845b.slice/crio-916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f WatchSource:0}: Error finding container 916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f: Status 404 returned error can't find the container with id 916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f
Nov 02 17:32:56.736736 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.736600    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c68c66_cc5c_4bc4_9937_691ece1b51b3.slice/crio-15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75 WatchSource:0}: Error finding container 15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75: Status 404 returned error can't find the container with id 15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75
Nov 02 17:32:56.769943 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.769096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:56.769943 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.769590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:56.774746 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.774507    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:32:56.774746 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.774651    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\""
Nov 02 17:32:56.774746 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.774709    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\""
Nov 02 17:32:56.780233 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.780107    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-622zn"]
Nov 02 17:32:56.782511 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.782484    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bxzmw"]
Nov 02 17:32:56.786126 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.784146    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ccbff4_2818_4e70_b2f4_a64c92f789e8.slice/crio-dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439 WatchSource:0}: Error finding container dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439: Status 404 returned error can't find the container with id dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439
Nov 02 17:32:56.802171 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.802146    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"]
Nov 02 17:32:56.807614 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.807571    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55feb08_7024_4795_9a42_7345567fe78b.slice/crio-1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207 WatchSource:0}: Error finding container 1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207: Status 404 returned error can't find the container with id 1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207
Nov 02 17:32:56.846774 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.846744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:56.846887 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.846805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:56.846887 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.846873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:56.846995 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.846912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.846995 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.846964    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:32:56.846995 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.846990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:56.847131 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847030    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847015266 +0000 UTC m=+31.573637046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:32:56.847131 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.847058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:56.847131 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.847085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:56.847131 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847117    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:56.847131 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847132    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847171    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847179    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847162907 +0000 UTC m=+31.573784690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847196    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847188214 +0000 UTC m=+31.573809994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847236    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847247    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847278    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847267013 +0000 UTC m=+31.573888799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:32:56.847371 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847346    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847336057 +0000 UTC m=+31.573957867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : configmap references non-existent config key: service-ca.crt
Nov 02 17:32:56.847648 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847404    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:32:56.847648 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847435    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847425062 +0000 UTC m=+31.574046844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:32:56.847648 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847470    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:56.847648 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.847501    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.847491663 +0000 UTC m=+31.574113447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:56.855914 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.855896    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"]
Nov 02 17:32:56.889098 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.889064    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"]
Nov 02 17:32:56.907793 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.907338    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gqgqg"]
Nov 02 17:32:56.911353 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.911316    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f693f58_c274_4b42_863d_7b4e73d8fb10.slice/crio-9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218 WatchSource:0}: Error finding container 9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218: Status 404 returned error can't find the container with id 9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218
Nov 02 17:32:56.915892 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.915865    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"]
Nov 02 17:32:56.922474 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.922439    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9f38d9_38b2_42e3_8236_a3fa68f311db.slice/crio-b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299 WatchSource:0}: Error finding container b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299: Status 404 returned error can't find the container with id b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299
Nov 02 17:32:56.922569 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.922527    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-hk6xf"]
Nov 02 17:32:56.923923 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:56.923897    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a1ad24_d5c4_49cf_aa71_b92f14918879.slice/crio-d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80 WatchSource:0}: Error finding container d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80: Status 404 returned error can't find the container with id d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80
Nov 02 17:32:56.948369 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:56.948340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:56.948474 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.948434    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:32:56.948519 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:56.948494    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:57.9484791 +0000 UTC m=+31.675100907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:32:57.092150 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.092111    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc" event={"ID":"be9f38d9-38b2-42e3-8236-a3fa68f311db","Type":"ContainerStarted","Data":"b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299"}
Nov 02 17:32:57.093652 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.093591    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bxzmw" event={"ID":"a893c12e-173e-4d9c-8080-8fa9992d9396","Type":"ContainerStarted","Data":"680fbb5ff09be8812473cd3fbc6900df9a29bb8437e418f1bf37965eb32293c7"}
Nov 02 17:32:57.094391 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.093953    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bxzmw" event={"ID":"a893c12e-173e-4d9c-8080-8fa9992d9396","Type":"ContainerStarted","Data":"33bd65d8c5af48f75b4e865db1cc4e591cb049145c0ef0baee2603f408034bdd"}
Nov 02 17:32:57.098560 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.098531    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" event={"ID":"17ccbff4-2818-4e70-b2f4-a64c92f789e8","Type":"ContainerStarted","Data":"dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439"}
Nov 02 17:32:57.099575 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.099557    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs" event={"ID":"b497b468-2062-4cc3-9687-0994f99979fd","Type":"ContainerStarted","Data":"8e759c85a4b26052307f7ac6277e6bb6ed7ebdb0b81d5c32dc6cdf08a5ab9f16"}
Nov 02 17:32:57.100592 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.100566    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hk6xf" event={"ID":"52a1ad24-d5c4-49cf-aa71-b92f14918879","Type":"ContainerStarted","Data":"d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80"}
Nov 02 17:32:57.102025 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.102004    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-892gj" event={"ID":"814ca1bd-35f4-4b88-9d95-f9e1cc47845b","Type":"ContainerStarted","Data":"fb061625e741ac727700e40332481ba22343d07c056974802d1079501e3251bb"}
Nov 02 17:32:57.102133 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.102045    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-892gj" event={"ID":"814ca1bd-35f4-4b88-9d95-f9e1cc47845b","Type":"ContainerStarted","Data":"916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f"}
Nov 02 17:32:57.103135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.103118    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gqgqg" event={"ID":"1f693f58-c274-4b42-863d-7b4e73d8fb10","Type":"ContainerStarted","Data":"9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218"}
Nov 02 17:32:57.104135 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.104118    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l" event={"ID":"d55feb08-7024-4795-9a42-7345567fe78b","Type":"ContainerStarted","Data":"1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207"}
Nov 02 17:32:57.105359 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.105333    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nwfc8" event={"ID":"f3c68c66-cc5c-4bc4-9937-691ece1b51b3","Type":"ContainerStarted","Data":"5d9b0b91e82cddeb58480bcc027fec54ca7f92f3074abfe36a9b79ddfbc632b3"}
Nov 02 17:32:57.105427 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.105375    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nwfc8" event={"ID":"f3c68c66-cc5c-4bc4-9937-691ece1b51b3","Type":"ContainerStarted","Data":"15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75"}
Nov 02 17:32:57.106314 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.106295    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm" event={"ID":"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c","Type":"ContainerStarted","Data":"9510ad10d82aa0ce9db50f8f4306ab5a72fe0ccee7ebd1972bf0093ca9563cb2"}
Nov 02 17:32:57.111159 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.111111    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bxzmw" podStartSLOduration=343.111094674 podStartE2EDuration="5m43.111094674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:57.110496081 +0000 UTC m=+30.837117895" watchObservedRunningTime="2026-11-02 17:32:57.111094674 +0000 UTC m=+30.837716478"
Nov 02 17:32:57.130469 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.130422    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nwfc8" podStartSLOduration=343.130407997 podStartE2EDuration="5m43.130407997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:57.129674445 +0000 UTC m=+30.856296248" watchObservedRunningTime="2026-11-02 17:32:57.130407997 +0000 UTC m=+30.857029795"
Nov 02 17:32:57.151193 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.151140    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-892gj" podStartSLOduration=343.151126816 podStartE2EDuration="5m43.151126816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:57.151013203 +0000 UTC m=+30.877635006" watchObservedRunningTime="2026-11-02 17:32:57.151126816 +0000 UTC m=+30.877748618"
Nov 02 17:32:57.764471 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.763584    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:57.768552 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.768505    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\""
Nov 02 17:32:57.768552 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.768544    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:32:57.856948 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.856906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.856974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.857029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.857054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.857114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.857142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.857163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.857303    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:32:57.857420 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.857375    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.857356466 +0000 UTC m=+33.583978261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:32:57.857824 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.857805    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:32:57.857897 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.857884    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.857866581 +0000 UTC m=+33.584488374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:32:57.857973 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.857957    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:57.858026 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.857977    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:32:57.858026 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858012    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.858000687 +0000 UTC m=+33.584622482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:32:57.858137 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858072    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:57.858137 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858084    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:32:57.858137 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858116    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.858105124 +0000 UTC m=+33.584726917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:32:57.858275 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858185    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.858172772 +0000 UTC m=+33.584794568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : configmap references non-existent config key: service-ca.crt
Nov 02 17:32:57.858275 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858246    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:57.858424 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858283    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.85826964 +0000 UTC m=+33.584891437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:57.858424 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858338    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:32:57.858424 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.858368    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.85835828 +0000 UTC m=+33.584980074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:32:57.958143 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:57.958104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:57.958434 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.958399    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:32:57.958516 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:57.958458    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:32:59.958439283 +0000 UTC m=+33.685061084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:32:59.120083 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.120032    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l" event={"ID":"d55feb08-7024-4795-9a42-7345567fe78b","Type":"ContainerStarted","Data":"97d1a94fdd214444dfbfd5ed03b99b2c9afcf0b6c21c04a3475e5ea802f06b96"}
Nov 02 17:32:59.139099 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.138036    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l" podStartSLOduration=289.234594751 podStartE2EDuration="4m51.13801698s" totalImagesPullingTime="1.903422229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:59.136287697 +0000 UTC m=+32.862909500" watchObservedRunningTime="2026-11-02 17:32:59.13801698 +0000 UTC m=+32.864638783"
Nov 02 17:32:59.576772 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.576699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:59.576772 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.576760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:32:59.577067 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.577056    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:32:59.577132 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.577119    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:31.577100893 +0000 UTC m=+65.303722691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : secret "metrics-daemon-secret" not found
Nov 02 17:32:59.586401 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.586369    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:59.592456 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.592250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:32:59.678890 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.678462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:59.682148 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.682092    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:59.831868 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.831765    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880823    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.880874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881037    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881053    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881111    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.881093035 +0000 UTC m=+37.607714829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881536    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881553    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881595    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.881580237 +0000 UTC m=+37.608202018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881662    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.881650582 +0000 UTC m=+37.608272367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : configmap references non-existent config key: service-ca.crt
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881722    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:59.882087 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881755    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.88174439 +0000 UTC m=+37.608366171 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:32:59.883233 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881805    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:32:59.883233 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881855    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.881828498 +0000 UTC m=+37.608450283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:32:59.883233 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881909    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:32:59.883233 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881946    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.881933843 +0000 UTC m=+37.608555626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:32:59.883233 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.881995    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:32:59.883233 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.882024    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.882014497 +0000 UTC m=+37.608636278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:32:59.921626 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.921596    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b26bx"]
Nov 02 17:32:59.929137 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:32:59.929107    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5188d247_bf34_437c_bde7_0abc73c312f5.slice/crio-e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4 WatchSource:0}: Error finding container e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4: Status 404 returned error can't find the container with id e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4
Nov 02 17:32:59.982017 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:32:59.981919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:32:59.982934 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.982617    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:32:59.983811 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:32:59.983090    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.983035679 +0000 UTC m=+37.709657481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:33:00.139447 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:00.139360    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b26bx" event={"ID":"5188d247-bf34-437c-bde7-0abc73c312f5","Type":"ContainerStarted","Data":"e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4"}
Nov 02 17:33:00.180491 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:00.180426    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pp7tz"]
Nov 02 17:33:00.186458 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:33:00.186410    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812e2a84_04de_4c0e_b8f1_f956381cb3fd.slice/crio-85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d WatchSource:0}: Error finding container 85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d: Status 404 returned error can't find the container with id 85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d
Nov 02 17:33:01.146624 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:01.146586    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pp7tz" event={"ID":"812e2a84-04de-4c0e-b8f1-f956381cb3fd","Type":"ContainerStarted","Data":"85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d"}
Nov 02 17:33:02.159922 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.159011    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm" event={"ID":"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c","Type":"ContainerStarted","Data":"37a51d35f174f94709456c92a2c901db773ac3e5a71812026af9cb88e2d15b46"}
Nov 02 17:33:02.162723 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.162695    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/0.log"
Nov 02 17:33:02.162909 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.162741    2566 generic.go:356] "Generic (PLEG): container finished" podID="17ccbff4-2818-4e70-b2f4-a64c92f789e8" containerID="cdc2db1d80320aeeeff69315b89bfd7b79be0863d938b871bf68eeb2c109e90a" exitCode=255
Nov 02 17:33:02.162909 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.162822    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" event={"ID":"17ccbff4-2818-4e70-b2f4-a64c92f789e8","Type":"ContainerDied","Data":"cdc2db1d80320aeeeff69315b89bfd7b79be0863d938b871bf68eeb2c109e90a"}
Nov 02 17:33:02.165034 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.163311    2566 scope.go:122] "RemoveContainer" containerID="cdc2db1d80320aeeeff69315b89bfd7b79be0863d938b871bf68eeb2c109e90a"
Nov 02 17:33:02.171865 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.171496    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hk6xf" event={"ID":"52a1ad24-d5c4-49cf-aa71-b92f14918879","Type":"ContainerStarted","Data":"eedb64b5917afc4e6302fa181807d401fcda8e9c3efc59d4771f0f461922f154"}
Nov 02 17:33:02.180454 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.180043    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm" podStartSLOduration=288.220070591 podStartE2EDuration="4m53.18002473s" totalImagesPullingTime="4.959954139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:02.178383664 +0000 UTC m=+35.905005467" watchObservedRunningTime="2026-11-02 17:33:02.18002473 +0000 UTC m=+35.906646534"
Nov 02 17:33:02.199898 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.198749    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-hk6xf" podStartSLOduration=288.314775608 podStartE2EDuration="4m53.198727503s" totalImagesPullingTime="4.883951895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:02.196958515 +0000 UTC m=+35.923580311" watchObservedRunningTime="2026-11-02 17:33:02.198727503 +0000 UTC m=+35.925349318"
Nov 02 17:33:02.452968 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.452520    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hffvc"]
Nov 02 17:33:02.478726 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.478683    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hffvc"]
Nov 02 17:33:02.478912 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.478749    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rx2qf-5jt9h"]
Nov 02 17:33:02.480073 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.480047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.485104 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.484987    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:33:02.485104 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.485037    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:33:02.485104 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.485081    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxnnv\""
Nov 02 17:33:02.503474 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.502994    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rx2qf-5jt9h"]
Nov 02 17:33:02.503474 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.503130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.614462 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.614619 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-data-volume\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.614619 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.614619 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95m9q\" (UniqueName: \"kubernetes.io/projected/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-api-access-95m9q\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.614619 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-crio-socket\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.614833 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.614833 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.614833 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.614833 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.614732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.715998 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.715962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-crio-socket\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.716111 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.716093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.716186 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.716121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.716186 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.716164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.716272 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.716188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.716887 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.716801    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.717193 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:02.717167    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:02.717301 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.717210    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-crio-socket\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.717301 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:02.717267    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.217242575 +0000 UTC m=+36.943864358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:02.717426 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.717314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.717426 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.717413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-data-volume\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.717520 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.717461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.717520 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.717488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95m9q\" (UniqueName: \"kubernetes.io/projected/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-api-access-95m9q\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.717615 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:02.717572    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:33:02.717663 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:02.717617    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:03.217606431 +0000 UTC m=+36.944228211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : configmap "kube-rbac-proxy" not found
Nov 02 17:33:02.717860 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.717811    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-data-volume\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.718116 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.718076    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.718565 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.718548    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.737423 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.737386    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.739571 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.739552    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95m9q\" (UniqueName: \"kubernetes.io/projected/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-api-access-95m9q\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:02.799888 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.799216    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"]
Nov 02 17:33:02.820892 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.807207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:02.820892 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.810165    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"]
Nov 02 17:33:02.820892 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.820632    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wb2z5\""
Nov 02 17:33:02.820892 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.820653    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:33:02.820892 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.820730    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:02.831407 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.831304    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"]
Nov 02 17:33:02.831407 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.831354    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"]
Nov 02 17:33:02.831626 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.831456    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:02.880588 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.880362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:02.922926 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:02.922662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ckt\" (UniqueName: \"kubernetes.io/projected/f8339155-7971-48b0-b048-babe5a75dc3e-kube-api-access-k6ckt\") pod \"migrator-66df5d496b-cp2bp\" (UID: \"f8339155-7971-48b0-b048-babe5a75dc3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:03.024976 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.024376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/a5742619-8be4-4c11-b8a3-abc18404e27b-kube-api-access-czh9g\") pod \"migrator-66df5d496b-hjh9z\" (UID: \"a5742619-8be4-4c11-b8a3-abc18404e27b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:03.024976 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.024500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ckt\" (UniqueName: \"kubernetes.io/projected/f8339155-7971-48b0-b048-babe5a75dc3e-kube-api-access-k6ckt\") pod \"migrator-66df5d496b-cp2bp\" (UID: \"f8339155-7971-48b0-b048-babe5a75dc3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:03.037448 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.037408    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ckt\" (UniqueName: \"kubernetes.io/projected/f8339155-7971-48b0-b048-babe5a75dc3e-kube-api-access-k6ckt\") pod \"migrator-66df5d496b-cp2bp\" (UID: \"f8339155-7971-48b0-b048-babe5a75dc3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:03.125592 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.125197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/a5742619-8be4-4c11-b8a3-abc18404e27b-kube-api-access-czh9g\") pod \"migrator-66df5d496b-hjh9z\" (UID: \"a5742619-8be4-4c11-b8a3-abc18404e27b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:03.137483 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.137453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:03.157638 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.157606    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/a5742619-8be4-4c11-b8a3-abc18404e27b-kube-api-access-czh9g\") pod \"migrator-66df5d496b-hjh9z\" (UID: \"a5742619-8be4-4c11-b8a3-abc18404e27b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:03.164658 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.164598    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:03.176220 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.176198    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rx2qf-5jt9h"]
Nov 02 17:33:03.178680 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.178651    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gqgqg" event={"ID":"1f693f58-c274-4b42-863d-7b4e73d8fb10","Type":"ContainerStarted","Data":"30b3e45b4e989f549cb63f07d945ce95e87a52714713339214f4e40126c51495"}
Nov 02 17:33:03.187889 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.187574    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:33:03.188239 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.188219    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/0.log"
Nov 02 17:33:03.188304 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.188262    2566 generic.go:356] "Generic (PLEG): container finished" podID="17ccbff4-2818-4e70-b2f4-a64c92f789e8" containerID="caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4" exitCode=255
Nov 02 17:33:03.188364 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.188346    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" event={"ID":"17ccbff4-2818-4e70-b2f4-a64c92f789e8","Type":"ContainerDied","Data":"caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4"}
Nov 02 17:33:03.188432 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.188401    2566 scope.go:122] "RemoveContainer" containerID="cdc2db1d80320aeeeff69315b89bfd7b79be0863d938b871bf68eeb2c109e90a"
Nov 02 17:33:03.189469 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.189185    2566 scope.go:122] "RemoveContainer" containerID="caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4"
Nov 02 17:33:03.189469 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.189411    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-622zn_openshift-console-operator(17ccbff4-2818-4e70-b2f4-a64c92f789e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" podUID="17ccbff4-2818-4e70-b2f4-a64c92f789e8"
Nov 02 17:33:03.193083 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.192863    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs" event={"ID":"b497b468-2062-4cc3-9687-0994f99979fd","Type":"ContainerStarted","Data":"4d5cccac9be678913fe2c10a3137fc35a5b2cd49811693d5939ba9df2086baa8"}
Nov 02 17:33:03.225787 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.221975    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs" podStartSLOduration=281.877519957 podStartE2EDuration="4m47.221956011s" totalImagesPullingTime="5.344436054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:03.220907474 +0000 UTC m=+36.947529278" watchObservedRunningTime="2026-11-02 17:33:03.221956011 +0000 UTC m=+36.948577814"
Nov 02 17:33:03.225787 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.222227    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gqgqg" podStartSLOduration=1.453089169 podStartE2EDuration="7.222219792s" totalImagesPullingTime="5.769130623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:03.200699304 +0000 UTC m=+36.927321108" watchObservedRunningTime="2026-11-02 17:33:03.222219792 +0000 UTC m=+36.948841598"
Nov 02 17:33:03.228853 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.228812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:03.229426 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.229406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:03.229992 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.229319    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:03.230190 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.230165    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:04.230144542 +0000 UTC m=+37.956766340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:03.230299 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.229639    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:33:03.230420 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.230410    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:04.230395492 +0000 UTC m=+37.957017291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : configmap "kube-rbac-proxy" not found
Nov 02 17:33:03.380005 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.379961    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s5jkp"]
Nov 02 17:33:03.405582 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.404103    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s5jkp"]
Nov 02 17:33:03.405582 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.404248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.408832 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.408575    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 17:33:03.408832 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.408678    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 17:33:03.408832 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.408692    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 17:33:03.408832 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.408798    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9fwwt\""
Nov 02 17:33:03.408832 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.408825    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 17:33:03.485421 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.485381    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"]
Nov 02 17:33:03.494016 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:33:03.490276    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8339155_7971_48b0_b048_babe5a75dc3e.slice/crio-f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e WatchSource:0}: Error finding container f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e: Status 404 returned error can't find the container with id f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e
Nov 02 17:33:03.513697 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.513668    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"]
Nov 02 17:33:03.517003 ip-10-0-135-5 kubenswrapper[2566]: W1102 17:33:03.516965    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5742619_8be4_4c11_b8a3_abc18404e27b.slice/crio-99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c WatchSource:0}: Error finding container 99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c: Status 404 returned error can't find the container with id 99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c
Nov 02 17:33:03.537946 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.537910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-key\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.538131 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.537962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.538251 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.538232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m4kx\" (UniqueName: \"kubernetes.io/projected/5dd0570a-e7e1-4681-937e-b357a6271f77-kube-api-access-5m4kx\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.639498 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.639401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4kx\" (UniqueName: \"kubernetes.io/projected/5dd0570a-e7e1-4681-937e-b357a6271f77-kube-api-access-5m4kx\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.639498 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.639466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-key\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.639712 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.639499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.640364 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.640329    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.643903 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.643654    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-key\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.649170 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.649145    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4kx\" (UniqueName: \"kubernetes.io/projected/5dd0570a-e7e1-4681-937e-b357a6271f77-kube-api-access-5m4kx\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.740470 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.740430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:03.943213 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:03.943373 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:03.943373 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:03.943373 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943344    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:33:03.943530 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943388    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:33:03.943530 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943403    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:33:03.943530 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943424    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.943403333 +0000 UTC m=+45.670025129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:33:03.943530 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943444    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.943434484 +0000 UTC m=+45.670056269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:33:03.943530 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:03.943530 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943521    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943542    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943554    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943590    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.943579022 +0000 UTC m=+45.670200802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943636    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.943626618 +0000 UTC m=+45.670248436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : configmap references non-existent config key: service-ca.crt
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943688    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943698    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943658    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.943647984 +0000 UTC m=+45.670269815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943726    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.9437165 +0000 UTC m=+45.670338289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:03.943756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943827    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:03.943875 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:03.943880    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:11.943869269 +0000 UTC m=+45.670491064 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:04.045027 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.044996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:04.045204 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.045150    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:33:04.045249 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.045208    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.045193906 +0000 UTC m=+45.771815699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:33:04.135801 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.135763    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-s5jkp"]
Nov 02 17:33:04.200378 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.199953    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp" event={"ID":"5dd0570a-e7e1-4681-937e-b357a6271f77","Type":"ContainerStarted","Data":"ed398d4ed3fe62b2e99f95cdd3f0fa17f14e714400f955b6d16cd497abbc0be1"}
Nov 02 17:33:04.211863 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.211809    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z" event={"ID":"a5742619-8be4-4c11-b8a3-abc18404e27b","Type":"ContainerStarted","Data":"99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c"}
Nov 02 17:33:04.214415 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.214379    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h" event={"ID":"8645d924-656d-4790-a0d1-511c42b60e2b","Type":"ContainerStarted","Data":"a000ce97a577eec15b230643f05b74f723441a9f4b77cac41053ebdb41c45d6b"}
Nov 02 17:33:04.214553 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.214436    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h" event={"ID":"8645d924-656d-4790-a0d1-511c42b60e2b","Type":"ContainerStarted","Data":"98afd59c2779f69fd8200f96335b8b7a082fffd490d75f000065b79232baa5e9"}
Nov 02 17:33:04.217307 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.217272    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc" event={"ID":"be9f38d9-38b2-42e3-8236-a3fa68f311db","Type":"ContainerStarted","Data":"608c7366760b1d47cd18366bf45f42dadcd8a16e3ccbc6c4299779c1b8aecf4d"}
Nov 02 17:33:04.220412 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.220389    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:33:04.220805 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.220787    2566 scope.go:122] "RemoveContainer" containerID="caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4"
Nov 02 17:33:04.221030 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.221006    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-622zn_openshift-console-operator(17ccbff4-2818-4e70-b2f4-a64c92f789e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" podUID="17ccbff4-2818-4e70-b2f4-a64c92f789e8"
Nov 02 17:33:04.223034 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.223014    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp" event={"ID":"f8339155-7971-48b0-b048-babe5a75dc3e","Type":"ContainerStarted","Data":"f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e"}
Nov 02 17:33:04.234034 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.233970    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h" podStartSLOduration=2.233950908 podStartE2EDuration="2.233950908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:04.231738196 +0000 UTC m=+37.958360000" watchObservedRunningTime="2026-11-02 17:33:04.233950908 +0000 UTC m=+37.960572711"
Nov 02 17:33:04.248065 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.247963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:04.248065 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.248036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:04.250278 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.250185    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:04.250278 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.250248    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:06.250228767 +0000 UTC m=+39.976850561 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:04.250526 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.250329    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:33:04.250526 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:04.250365    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:06.250354163 +0000 UTC m=+39.976975971 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : configmap "kube-rbac-proxy" not found
Nov 02 17:33:04.271067 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:04.270511    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc" podStartSLOduration=282.333877447 podStartE2EDuration="4m49.27049801s" totalImagesPullingTime="6.936620563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:04.268891407 +0000 UTC m=+37.995513211" watchObservedRunningTime="2026-11-02 17:33:04.27049801 +0000 UTC m=+37.997119811"
Nov 02 17:33:05.228199 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:05.228129    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pp7tz" event={"ID":"812e2a84-04de-4c0e-b8f1-f956381cb3fd","Type":"ContainerStarted","Data":"99541fed482e150cd07bc5ac84c28dfd457c9a2e681877be7eb6fe09c47ce4e0"}
Nov 02 17:33:05.228663 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:05.228219    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:33:05.230198 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:05.230169    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp" event={"ID":"5dd0570a-e7e1-4681-937e-b357a6271f77","Type":"ContainerStarted","Data":"7029b1c4b25ab2edfa85ac50ae5df6a0217d1b9cd306a519cf26260d268c3760"}
Nov 02 17:33:05.244664 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:05.244610    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pp7tz" podStartSLOduration=34.27171146 podStartE2EDuration="38.244592945s" totalImagesPullingTime="3.972881485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:05.243948688 +0000 UTC m=+38.970570537" watchObservedRunningTime="2026-11-02 17:33:05.244592945 +0000 UTC m=+38.971214749"
Nov 02 17:33:05.260519 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:05.260460    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp" podStartSLOduration=2.260445729 podStartE2EDuration="2.260445729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:05.258787675 +0000 UTC m=+38.985409488" watchObservedRunningTime="2026-11-02 17:33:05.260445729 +0000 UTC m=+38.987067531"
Nov 02 17:33:06.236612 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.236099    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z" event={"ID":"a5742619-8be4-4c11-b8a3-abc18404e27b","Type":"ContainerStarted","Data":"e66fc9eb8279d18e613b89f68869f4e7f42d9300370ab02012f5f97a0f66f332"}
Nov 02 17:33:06.238928 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.238891    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp" event={"ID":"f8339155-7971-48b0-b048-babe5a75dc3e","Type":"ContainerStarted","Data":"917b5c77d4367c990d192526a6bebd4d9549f906ee242fc972e4fdd7429eb2db"}
Nov 02 17:33:06.254117 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.254045    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z" podStartSLOduration=2.172168573 podStartE2EDuration="4.254024914s" totalImagesPullingTime="2.081856341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:06.251949537 +0000 UTC m=+39.978571340" watchObservedRunningTime="2026-11-02 17:33:06.254024914 +0000 UTC m=+39.980646718"
Nov 02 17:33:06.266358 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.266309    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp" podStartSLOduration=2.161963706 podStartE2EDuration="4.266294146s" totalImagesPullingTime="2.10433044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:06.265887666 +0000 UTC m=+39.992509469" watchObservedRunningTime="2026-11-02 17:33:06.266294146 +0000 UTC m=+39.992915948"
Nov 02 17:33:06.274284 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.273485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:06.274284 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.273553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:06.274284 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:06.273744    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:06.274284 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:06.273750    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:33:06.274284 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:06.273798    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.273781946 +0000 UTC m=+44.000403728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:06.274284 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:06.273817    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.27380765 +0000 UTC m=+44.000429431 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : configmap "kube-rbac-proxy" not found
Nov 02 17:33:06.465461 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.465423    2566 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:06.465616 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.465499    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:06.465929 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:06.465915    2566 scope.go:122] "RemoveContainer" containerID="caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4"
Nov 02 17:33:06.466147 ip-10-0-135-5 kubenswrapper[2566]: E1102 17:33:06.466120    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-622zn_openshift-console-operator(17ccbff4-2818-4e70-b2f4-a64c92f789e8)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" podUID="17ccbff4-2818-4e70-b2f4-a64c92f789e8"
Nov 02 17:33:07.247085 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:07.247036    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b26bx" event={"ID":"5188d247-bf34-437c-bde7-0abc73c312f5","Type":"ContainerStarted","Data":"a1ffce4a2ebadbb290de60c0843c760d6d9ba3c984f8d6bf6e2407752efa693c"}
Nov 02 17:33:07.263560 ip-10-0-135-5 kubenswrapper[2566]: I1102 17:33:07.263508    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b26bx" podStartSLOduration=33.166236354 podStartE2EDuration="40.2634934s" totalImagesPullingTime="7.097257046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:07.2617323 +0000 UTC m=+40.988354104" watchObservedRunningTime="2026-11-02 17:33:07.2634934 +0000 UTC m=+40.990115241"
Nov 02 17:33:07.348264 ip-10-0-135-5 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 17:33:07.358291 ip-10-0-135-5 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 17:33:07.358469 ip-10-0-135-5 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 17:33:07.358519 ip-10-0-135-5 systemd[1]: kubelet.service: Consumed 3.409s CPU time.
Nov 02 17:33:07.370594 ip-10-0-135-5 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:33:07.452513 ip-10-0-135-5 kubenswrapper[5042]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:33:07.452513 ip-10-0-135-5 kubenswrapper[5042]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:33:07.452513 ip-10-0-135-5 kubenswrapper[5042]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:33:07.452513 ip-10-0-135-5 kubenswrapper[5042]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454538    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454633    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454637    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454640    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454642    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454645    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.454638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454648    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454652    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454654    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454657    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454659    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454662    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454664    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454668    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454671    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454673    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454676    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454678    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454681    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454684    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454694    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454698    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454700    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454703    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454706    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454709    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.454925 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454712    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454715    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454717    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454720    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454722    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454725    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454727    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454732    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454736    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454739    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454742    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454745    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454747    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454749    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454752    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454754    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454757    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454759    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454762    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454764    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.455415 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454767    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454769    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454772    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454775    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454777    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454780    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454782    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454785    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454787    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454790    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454792    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454795    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454797    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454800    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454802    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454805    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454808    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454812    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454815    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.455937 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454833    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454850    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454853    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454856    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454859    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454862    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454864    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454867    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454869    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454872    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454875    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454877    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454880    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454882    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454890    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454893    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454895    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454898    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454901    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454903    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.456398 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454906    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454909    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454912    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454915    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454917    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454920    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454923    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454926    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454929    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454932    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454935    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454938    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454941    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454944    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454947    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454950    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454953    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454956    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454958    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.456907 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454961    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.454963    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455778    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455785    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455788    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455791    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455794    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455796    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455799    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455801    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455804    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455807    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455810    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455812    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455815    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455817    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455820    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455822    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455825    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455828    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.457403 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455830    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455833    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455852    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455855    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455857    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455860    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455863    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455866    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455872    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455875    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455877    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455880    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455883    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455886    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455889    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455891    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455894    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455896    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455899    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455901    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.457917 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455904    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455906    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455909    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455911    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455914    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455916    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455919    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455921    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455924    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455927    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455929    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455934    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455938    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455941    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455944    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455946    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455949    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455952    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455954    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.458457 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455957    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455961    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455964    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455966    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455969    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455971    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455974    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455977    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455980    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455983    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455985    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455988    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455990    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455993    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455995    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.455998    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456001    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456003    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456006    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.458933 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456008    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456011    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456015    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456019    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456023    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456026    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456028    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456031    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456033    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456036    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456038    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456041    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456044    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456046    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456050    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456054    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456056    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456061    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456063    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.459395 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456066    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456068    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456071    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456074    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456076    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456079    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456081    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456084    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456086    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456088    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456091    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456591    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456598    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456602    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456605    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456609    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456612    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456614    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456617    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.459875 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456619    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456622    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456625    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456627    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456630    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456632    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456635    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456638    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456644    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456653    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456657    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456659    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456662    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456665    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456668    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456671    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456673    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456676    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456679    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.460451 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456681    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456684    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456686    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456689    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456691    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456694    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456697    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456699    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456702    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456704    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456707    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456709    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456712    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456715    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456717    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456720    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456722    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456725    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456727    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.461006 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456729    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456732    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456737    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456740    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456743    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456745    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456748    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456750    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456753    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456756    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456759    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456761    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456764    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456766    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456769    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456771    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456774    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456776    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456779    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456781    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.461517 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456784    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456787    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456789    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456792    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456794    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456797    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456799    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456803    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456805    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456808    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456810    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456813    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456815    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456818    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456822    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456824    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456827    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456830    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456832    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456851    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.462277 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456854    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456856    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456859    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456862    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456864    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456867    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456869    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456872    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456874    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456877    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456879    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456882    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456885    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456888    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456891    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456894    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456896    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456898    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456901    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.456904    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.463011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.456997    5042 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457012    5042 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457018    5042 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457023    5042 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457027    5042 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457031    5042 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457039    5042 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457044    5042 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457047    5042 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457050    5042 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457054    5042 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457057    5042 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457062    5042 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457065    5042 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457068    5042 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457071    5042 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457074    5042 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457077    5042 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457081    5042 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457083    5042 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457086    5042 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457089    5042 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457093    5042 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457097    5042 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:33:07.463494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457100    5042 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457103    5042 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457106    5042 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457110    5042 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457113    5042 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457116    5042 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457119    5042 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457122    5042 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457126    5042 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457129    5042 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457131    5042 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457134    5042 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457137    5042 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457140    5042 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457144    5042 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457148    5042 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457151    5042 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457154    5042 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457157    5042 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457161    5042 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457164    5042 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457168    5042 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457170    5042 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457174    5042 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457176    5042 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:33:07.464079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457179    5042 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457183    5042 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457186    5042 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457189    5042 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457192    5042 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457196    5042 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457199    5042 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457202    5042 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457206    5042 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457209    5042 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457212    5042 flags.go:64] FLAG: --help="false"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457215    5042 flags.go:64] FLAG: --hostname-override="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457218    5042 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457221    5042 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457223    5042 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457227    5042 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457230    5042 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457233    5042 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457235    5042 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457238    5042 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457241    5042 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457244    5042 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457247    5042 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457250    5042 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:33:07.464694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457253    5042 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457255    5042 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457258    5042 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457262    5042 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457265    5042 flags.go:64] FLAG: --lock-file=""
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457268    5042 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457271    5042 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457274    5042 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457283    5042 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457286    5042 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457289    5042 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457292    5042 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457295    5042 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457299    5042 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457301    5042 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457304    5042 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457308    5042 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457311    5042 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457315    5042 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457318    5042 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457321    5042 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457325    5042 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457328    5042 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457339    5042 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457343    5042 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:33:07.465289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457348    5042 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457356    5042 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457359    5042 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457362    5042 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457365    5042 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457368    5042 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457371    5042 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457374    5042 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457376    5042 flags.go:64] FLAG: --port="10250"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457379    5042 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457382    5042 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-055f47dd784e84826"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457386    5042 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457389    5042 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457392    5042 flags.go:64] FLAG: --register-node="true"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457395    5042 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457399    5042 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457402    5042 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457405    5042 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457407    5042 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457412    5042 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457415    5042 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457418    5042 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457421    5042 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457424    5042 flags.go:64] FLAG: --runonce="false"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457427    5042 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457430    5042 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457433    5042 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:33:07.465926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457436    5042 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457439    5042 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457442    5042 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457445    5042 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457448    5042 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457453    5042 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457456    5042 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457459    5042 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457461    5042 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457464    5042 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457467    5042 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457470    5042 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457476    5042 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457479    5042 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457482    5042 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457485    5042 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457489    5042 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457492    5042 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457495    5042 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457498    5042 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457501    5042 flags.go:64] FLAG: --v="2"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457505    5042 flags.go:64] FLAG: --version="false"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457509    5042 flags.go:64] FLAG: --vmodule=""
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457513    5042 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457517    5042 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:33:07.466598 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457658    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457662    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457666    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457668    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457671    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457674    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457679    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457681    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457684    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457686    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457689    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457692    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457696    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457699    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457701    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457704    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457706    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457709    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457711    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457714    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.467262 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457716    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457719    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457721    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457724    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457726    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457730    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457733    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457735    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457738    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457740    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457743    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457745    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457748    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457750    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457752    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457755    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457758    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457760    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457764    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.467735 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457767    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457769    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457772    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457774    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457776    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457780    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457784    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457788    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457791    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457795    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457798    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457801    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457803    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457806    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457808    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457811    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457813    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457816    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457818    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.468374 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457821    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457824    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457826    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457829    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457831    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457833    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457852    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457855    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457857    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457860    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457863    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457865    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457869    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457871    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457874    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457876    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457879    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457881    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457885    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457888    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.469038 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457890    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457893    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457896    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457898    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457901    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457903    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457906    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457908    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457911    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457914    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457916    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457919    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457921    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457924    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457927    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457930    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457933    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457935    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457938    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.469726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457940    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457943    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457945    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457947    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457950    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457954    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457956    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457959    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.457961    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.457987    5042 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458025    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458030    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458034    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458037    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458040    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458042    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.470485 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458045    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458047    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458050    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458052    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458055    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458057    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458060    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458062    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458065    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458067    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458070    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458072    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458075    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458077    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458080    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458082    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458085    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458088    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458090    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.471233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458093    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458095    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458099    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458102    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458121    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458123    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458126    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458128    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458130    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458134    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458137    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458139    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458142    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458145    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458147    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458150    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458153    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458155    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458157    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458160    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.471902 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458163    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458165    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458168    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458170    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458173    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458176    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458179    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458181    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458184    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458186    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458189    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458191    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458196    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458200    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458204    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458208    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458212    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458214    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458217    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.472576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458220    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458222    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458228    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458230    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458233    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458235    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458238    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458241    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458243    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458245    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458248    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458250    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458252    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458255    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458258    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458260    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458262    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458265    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458267    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458270    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.473291 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458273    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458276    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458278    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458281    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458283    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458286    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458288    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458293    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458295    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458298    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458301    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458304    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458306    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458308    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458312    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458315    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458317    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458320    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458322    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.473773 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458325    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458327    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.458330    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.463722    5042 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.463741    5042 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.463761    5042 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.463766    5042 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463829    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463834    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463856    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463859    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463862    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463865    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463868    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463871    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463873    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463876    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463879    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463881    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.474251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463884    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463887    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463889    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463892    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463895    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463899    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463904    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463908    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463912    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463916    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463920    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463925    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463930    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463933    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463937    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463940    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463943    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463946    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463949    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463952    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.474697 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463954    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463957    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463959    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463962    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463964    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463967    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463970    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463972    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463975    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463978    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463980    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463983    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463986    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463988    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463991    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463994    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463996    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.463999    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464002    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464004    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.475198 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464007    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464009    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464012    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464015    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464018    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464020    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464023    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464026    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464028    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464032    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464034    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464037    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464040    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464043    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464045    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464048    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464050    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464053    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464055    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.475666 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464058    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464060    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464063    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464066    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464068    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464071    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464073    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464077    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464082    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464085    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464088    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464090    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464093    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464096    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464098    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464101    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464104    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464106    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464109    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464112    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.476139 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464115    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464117    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464120    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464123    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464125    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464129    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464134    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464137    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464140    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464142    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464144    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464147    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464149    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464152    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464154    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.476646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.464185    5042 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464329    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464334    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464337    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464340    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464342    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464347    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464349    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464352    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464355    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464357    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464360    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464362    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464365    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464368    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464370    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464373    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464377    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464379    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464381    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.477124 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464384    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464387    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464390    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464392    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464395    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464397    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464400    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464402    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464405    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464408    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464410    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464412    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464415    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464418    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464420    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464423    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464425    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464428    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464430    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464434    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.477608 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464436    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464439    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464441    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464444    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464446    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464449    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464451    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464453    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464456    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464459    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464461    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464465    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464468    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464471    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464474    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464477    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464480    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464482    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464485    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.478171 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464487    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464491    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464495    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464498    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464501    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464503    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464506    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464508    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464510    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464513    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464515    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464518    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464520    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464524    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464527    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464529    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464531    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464534    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464536    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.478639 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464539    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464541    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464544    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464546    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464549    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464551    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464554    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464557    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464559    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464561    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464564    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464566    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464569    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464571    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464574    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464576    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464579    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464581    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464583    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464586    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.479122 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464588    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464591    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464594    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464597    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464599    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464601    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464604    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464615    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464618    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.464642    5042 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464674    5042 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464680    5042 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464683    5042 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464686    5042 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464689    5042 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:33:07.479814 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464692    5042 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464694    5042 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464698    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464700    5042 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464703    5042 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464706    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464708    5042 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464711    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464714    5042 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464716    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464718    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464721    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464723    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464726    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464728    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464730    5042 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464733    5042 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464735    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464738    5042 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464740    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:33:07.480486 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464743    5042 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464745    5042 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464748    5042 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464753    5042 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464756    5042 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464764    5042 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464767    5042 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464770    5042 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464772    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464775    5042 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464777    5042 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464780    5042 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464782    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464785    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464787    5042 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464790    5042 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464792    5042 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464794    5042 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464797    5042 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464799    5042 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:33:07.481174 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464802    5042 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464804    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464807    5042 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464809    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464812    5042 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464814    5042 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464817    5042 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464819    5042 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464822    5042 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464824    5042 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464827    5042 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464829    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464831    5042 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464834    5042 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464850    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464854    5042 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464856    5042 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464859    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464862    5042 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464865    5042 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:33:07.481723 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464868    5042 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464870    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464873    5042 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464875    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464878    5042 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464882    5042 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464884    5042 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464887    5042 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464889    5042 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464892    5042 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464895    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464898    5042 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464900    5042 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464903    5042 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464905    5042 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464908    5042 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464910    5042 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464913    5042 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464915    5042 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464918    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:33:07.482275 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464920    5042 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464923    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464925    5042 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464927    5042 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464930    5042 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464933    5042 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464935    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464939    5042 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464942    5042 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464944    5042 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464947    5042 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464951    5042 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464955    5042 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464960    5042 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464963    5042 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464966    5042 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464970    5042 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464974    5042 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464978    5042 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:33:07.482766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464980    5042 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:07.464983    5042 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.465062    5042 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466118    5042 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466185    5042 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466363    5042 server.go:1014] "Starting client certificate rotation"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466503    5042 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466551    5042 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:27:26 +0000 UTC" deadline="2028-04-19 01:31:25.445316286 +0000 UTC"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466584    5042 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12799h58m17.97873394s"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.466819    5042 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.467938    5042 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.469527    5042 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.471809    5042 log.go:25] "Validated CRI v1 image API"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.472681    5042 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:33:07.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.475216    5042 fs.go:135] Filesystem UUIDs: map[1ea59125-eb7b-49a5-8301-b04cf2db486b:/dev/nvme0n1p4 77583fc4-01b8-4c9e-a019-1735d6f8b2c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Nov 02 17:33:07.483617 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.475233    5042 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33bd65d8c5af48f75b4e865db1cc4e591cb049145c0ef0baee2603f408034bdd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33bd65d8c5af48f75b4e865db1cc4e591cb049145c0ef0baee2603f408034bdd/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e759c85a4b26052307f7ac6277e6bb6ed7ebdb0b81d5c32dc6cdf08a5ab9f16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e759c85a4b26052307f7ac6277e6bb6ed7ebdb0b81d5c32dc6cdf08a5ab9f16/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9510ad10d82aa0ce9db50f8f4306ab5a72fe0ccee7ebd1972bf0093ca9563cb2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9510ad10d82aa0ce9db50f8f4306ab5a72fe0ccee7ebd1972bf0093ca9563cb2/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98afd59c2779f69fd8200f96335b8b7a082fffd490d75f000065b79232baa5e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98afd59c2779f69fd8200f96335b8b7a082fffd490d75f000065b79232baa5e9/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e/userdata/shm major:0 minor:55 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed398d4ed3fe62b2e99f95cdd3f0fa17f14e714400f955b6d16cd497abbc0be1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed398d4ed3fe62b2e99f95cdd3f0fa17f14e714400f955b6d16cd497abbc0be1/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/032f9f33-d723-46ad-9d89-45956596eec5/volumes/kubernetes.io~projected/kube-api-access-q6tjt:{mountpoint:/var/lib/kubelet/pods/032f9f33-d723-46ad-9d89-45956596eec5/volumes/kubernetes.io~projected/kube-api-access-q6tjt major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17ccbff4-2818-4e70-b2f4-a64c92f789e8/volumes/kubernetes.io~projected/kube-api-access-nq28j:{mountpoint:/var/lib/kubelet/pods/17ccbff4-2818-4e70-b2f4-a64c92f789e8/volumes/kubernetes.io~projected/kube-api-access-nq28j major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17ccbff4-2818-4e70-b2f4-a64c92f789e8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/17ccbff4-2818-4e70-b2f4-a64c92f789e8/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f693f58-c274-4b42-863d-7b4e73d8fb10/volumes/kubernetes.io~projected/kube-api-access-xjqh9:{mountpoint:/var/lib/kubelet/pods/1f693f58-c274-4b42-863d-7b4e73d8fb10/volumes/kubernetes.io~projected/kube-api-access-xjqh9 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f693f58-c274-4b42-863d-7b4e73d8fb10/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1f693f58-c274-4b42-863d-7b4e73d8fb10/volumes/kubernetes.io~secret/cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/volumes/kubernetes.io~projected/kube-api-access-xqcnt:{mountpoint:/var/lib/kubelet/pods/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/volumes/kubernetes.io~projected/kube-api-access-xqcnt major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volumes/kubernetes.io~projected/kube-api-access-89ld7:{mountpoint:/var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volumes/kubernetes.io~projected/kube-api-access-89ld7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/505743f8-9737-44a8-987b-62f5d35212d3/volumes/kubernetes.io~projected/kube-api-access-87zcz:{mountpoint:/var/lib/kubelet/pods/505743f8-9737-44a8-987b-62f5d35212d3/volumes/kubernetes.io~projected/kube-api-access-87zcz major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5188d247-bf34-437c-bde7-0abc73c312f5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5188d247-bf34-437c-bde7-0abc73c312f5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a1ad24-d5c4-49cf-aa71-b92f14918879/volumes/kubernetes.io~projected/kube-api-access-fwm2f:{mountpoint:/var/lib/kubelet/pods/52a1ad24-d5c4-49cf-aa71-b92f14918879/volumes/kubernetes.io~projected/kube-api-access-fwm2f major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52a1ad24-d5c4-49cf-aa71-b92f14918879/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/52a1ad24-d5c4-49cf-aa71-b92f14918879/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/565006a5-a1dd-4383-8078-f99d66892a75/volumes/kubernetes.io~projected/kube-api-access-vls44:{mountpoint:/var/lib/kubelet/pods/565006a5-a1dd-4383-8078-f99d66892a75/volumes/kubernetes.io~projected/kube-api-access-vls44 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~projected/kube-api-access-6lbkf:{mountpoint:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~projected/kube-api-access-6lbkf major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd0570a-e7e1-4681-937e-b357a6271f77/volumes/kubernetes.io~projected/kube-api-access-5m4kx:{mountpoint:/var/lib/kubelet/pods/5dd0570a-e7e1-4681-937e-b357a6271f77/volumes/kubernetes.io~projected/kube-api-access-5m4kx major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5dd0570a-e7e1-4681-937e-b357a6271f77/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5dd0570a-e7e1-4681-937e-b357a6271f77/volumes/kubernetes.io~secret/signing-key major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63048823-3df5-48fa-a65d-c1d789a319c9/volumes/kubernetes.io~projected/kube-api-access-wc8vv:{mountpoint:/var/lib/kubelet/pods/63048823-3df5-48fa-a65d-c1d789a319c9/volumes/kubernetes.io~projected/kube-api-access-wc8vv major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~projected/kube-api-access-ncq4r:{mountpoint:/var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~projected/kube-api-access-ncq4r major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72ce72d3-fe26-4612-ac57-078f9f7bbfbe/volumes/kubernetes.io~projected/kube-api-access-95m9q:{mountpoint:/var/lib/kubelet/pods/72ce72d3-fe26-4612-ac57-078f9f7bbfbe/volumes/kubernetes.io~projected/kube-api-access-95m9q major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~projected/kube-api-access-p8dq8:{mountpoint:/var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~projected/kube-api-access-p8dq8 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e1fc379-cd57-4b14-be52-b3975a5f6514/volumes/kubernetes.io~projected/kube-api-access-56zzf:{mountpoint:/var/lib/kubelet/pods/7e1fc379-cd57-4b14-be52-b3975a5f6514/volumes/kubernetes.io~projected/kube-api-access-56zzf major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/812e2a84-04de-4c0e-b8f1-f956381cb3fd/volumes/kubernetes.io~projected/kube-api-access-hwsfg:{mountpoint:/var/lib/kubelet/pods/812e2a84-04de-4c0e-b8f1-f956381cb3fd/volumes/kubernetes.io~projected/kube-api-access-hwsfg major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/814ca1bd-35f4-4b88-9d95-f9e1cc47845b/volumes/kubernetes.io~projected/kube-api-access-bczck:{mountpoint:/var/lib/kubelet/pods/814ca1bd-35f4-4b88-9d95-f9e1cc47845b/volumes/kubernetes.io~projected/kube-api-access-bczck major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8645d924-656d-4790-a0d1-511c42b60e2b/volumes/kubernetes.io~projected/kube-api-access-rgfbs:{mountpoint:/var/lib/kubelet/pods/8645d924-656d-4790-a0d1-511c42b60e2b/volumes/kubernetes.io~projected/kube-api-access-rgfbs major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~projected/kube-api-access-599dg:{mountpoint:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~projected/kube-api-access-599dg major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5742619-8be4-4c11-b8a3-abc18404e27b/volumes/kubernetes.io~projected/kube-api-access-czh9g:{mountpoint:/var/lib/kubelet/pods/a5742619-8be4-4c11-b8a3-abc18404e27b/volumes/kubernetes.io~projected/kube-api-access-czh9g major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a893c12e-173e-4d9c-8080-8fa9992d9396/volumes/kubernetes.io~projected/kube-api-access-6xszp:{mountpoint:/var/lib/kubelet/pods/a893c12e-173e-4d9c-8080-8fa9992d9396/volumes/kubernetes.io~projected/kube-api-access-6xszp major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b497b468-2062-4cc3-9687-0994f99979fd/volumes/kubernetes.io~projected/kube-api-access-6j4x8:{mountpoint:/var/lib/kubelet/pods/b497b468-2062-4cc3-9687-0994f99979fd/volumes/kubernetes.io~projected/kube-api-access-6j4x8 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b497b468-2062-4cc3-9687-0994f99979fd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b497b468-2062-4cc3-9687-0994f99979fd/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b90ebafb-a030-43ce-a32a-edc04ce61c09/volumes/kubernetes.io~projected/kube-api-access-t42w6:{mountpoint:/var/lib/kubelet/pods/b90ebafb-a030-43ce-a32a-edc04ce61c09/volumes/kubernetes.io~projected/kube-api-access-t42w6 major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be9f38d9-38b2-42e3-8236-a3fa68f311db/volumes/kubernetes.io~projected/kube-api-access-jcm2m:{mountpoint:/var/lib/kubelet/pods/be9f38d9-38b2-42e3-8236-a3fa68f311db/volumes/kubernetes.io~projected/kube-api-access-jcm2m major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c58d6490-3f26-48cc-95eb-440097888b15/volumes/kubernetes.io~projected/kube-api-access-47sj4:{mountpoint:/var/lib/kubelet/pods/c58d6490-3f26-48cc-95eb-440097888b15/volumes/kubernetes.io~projected/kube-api-access-47sj4 major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e48a54-0431-4e67-a886-30ce38fc01ef/volumes/kubernetes.io~projected/kube-api-access-6m2n6:{mountpoint:/var/lib/kubelet/pods/d4e48a54-0431-4e67-a886-30ce38fc01ef/volumes/kubernetes.io~projected/kube-api-access-6m2n6 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d55feb08-7024-4795-9a42-7345567fe78b/volumes/kubernetes.io~projected/kube-api-access-99htf:{mountpoint:/var/lib/kubelet/pods/d55feb08-7024-4795-9a42-7345567fe78b/volumes/kubernetes.io~projected/kube-api-access-99htf major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d806b75d-88ca-4e51-94c3-cd45fd8888db/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d806b75d-88ca-4e51-94c3-cd45fd8888db/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce/volumes/kubernetes.io~projected/kube-api-access-mmqbq:{mountpoint:/var/lib/kubelet/pods/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce/volumes/kubernetes.io~projected/kube-api-access-mmqbq major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3c68c66-cc5c-4bc4-9937-691ece1b51b3/volumes/kubernetes.io~projected/kube-api-access-hxjlz:{mountpoint:/var/lib/kubelet/pods/f3c68c66-cc5c-4bc4-9937-691ece1b51b3/volumes/kubernetes.io~projected/kube-api-access-hxjlz major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8339155-7971-48b0-b048-babe5a75dc3e/volumes/kubernetes.io~projected/kube-api-access-k6ckt:{mountpoint:/var/lib/kubelet/pods/f8339155-7971-48b0-b048-babe5a75dc3e/volumes/kubernetes.io~projected/kube-api-access-k6ckt major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd30ffd8-a9b2-46a3-b057-e25e98540942/volumes/kubernetes.io~projected/kube-api-access-rzhxr:{mountpoint:/var/lib/kubelet/pods/fd30ffd8-a9b2-46a3-b057-e25e98540942/volumes/kubernetes.io~projected/kube-api-access-rzhxr major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/435763ba9a4e9c9aa20345ccf1ba0ea0834f8cba92d2e2e2ccaf67c914a2ab82/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/05f0d9987c79f145feadca9c37ae371e6e8efa846ff74e998e7bf32e07914c47/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/446503dd56f17eb575285121ee2de14f54e3e5ade46c253ef2e1848ad86a64bf/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d6e223e2183efbb07c72acd1ac0bab67e9a5c3c8a53e2bdc0fc656138065f274/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b7e1cad22a324c990247e3f706a08eab5dd80c3b3bf8c031ed66aed61cd444c6/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f6f64aee082839fbb193222a9b1f63a2a25c06e416d932fcb4efe2e4ad4d4035/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5154e3f2008911a249a9146f1d8cd5862ba7a48b0448c5d7846ff1f857dd36d5/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cae7a277dd59478fcb051795eb7407d6cd630d30795250c8d59b3ea75b52c15c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3268dfe19d895ef5f5f334b43769a52e1bc204984e4434a5748c8fbb4abecb13/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d25c6ba80edee4a2ec826cfc7e887c2596c7a6caec1bf6cafebf0e272ca0d157/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/80b5010a5cabe744189eb069de5c11336f7a92a065ad43508b3b8b1a7272d144/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/69aa3824673ecdf36d3750e364ec0e2acbe6c6b028ea9a8e4684b0b12175d5fc/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/937e617cabee7e4409b5e5f859d908dd1fcc7b1db1d0529aa630691353091dda/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/b2623c993bf72a48e54eaeca168e8cf2e3a66ad16d43d446dcb7f366e23ea62b/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/3a1976191a170efb0c117cd9f72bff39c0e0bd1acde87fc2096b6382ddabd56b/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9003669cae654f56c24bd9d428dcdc9ee93ea3d7e6abfc9e918049650f80a7cd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/056c75031db7e454375ea792c79cff390829f71eb9754023c2c2830e284b1c7c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c1078c05467d1797d3c597f0cd13cde2fd2831386453782efc995c6900478680/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/92c82a58a2694ca6dae718cd36c8f9617a9c622327ed5da360fd5b8748f1ae24/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a4729a0e0e910630eded1b5fe15d380f5a498a027eb51a5fea57ae49f9bbbce0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d2c2a264f4271094448900668b2a9805ba714b700485efb5b62fc78d2b0d9e44/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9984f0fac0a0e2dca702bd8dfe26097b94b0db8392a76d3421c9b5e896b47a57/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/97f8480a3dcd592c9b9f171ef6c8a68fefac23b33bdf32158f9f34c947ead860/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/0f1f9da26eaf9cd01f2346d2dbf6fd2d435c3bb94e1661b0ca21c39a9497a1c4/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/2981e000ee5301251355dbd9c7bef411af2975d7ae8b01cd5dcd7e1cf99e42a8/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0087e19942f04208e8c6407cab8268c66f35cefb69c567a6189e36dcbb454887/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/b620fd437259b4aca535a7e446cdd61b1f08a1bc08086ab0313e54a250fea3f5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/07d51597f6ed00558c6971c9a6a0227fee7beeee30959bebcdc406790ef41a97/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e24ad4cb1f39dda3a3e330bf963aaf2c8175d36410a1a63f0e216fd304fd29a7/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/5d61aaf0f39cf3d91290d87edcfe709145aa1fe0b4482d81bb52a14574c98b76/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/fda373a58938936cc66a352bdcfd8aee43b25041bb4929416c58be53b62d46ce/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/8fc1d542d95fc8dd8e82320ef909c05c7ad52941ee1498538707a32ffe4453ab/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b7a1c2bc41214b78bf54b79835ddaf9921b08ba461731c8e5381c46da33c434d/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/5f7cfc10492593c498c35e416de034a6e76ad6a3a21edcb2f2ccf11fca303148/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/bfdb08ee89d5d466955a69936fafb78480a993496f5b8e07138b405f0ff2f01e/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/dadc27f398c6d59fcefba45ecb472f81971aa74c06cd262b246bddbc49859222/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/88c8a6b1648b192e91518429eebcee281de8631639e563a08dca89ff57c8d0a8/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c4e503b1c26fbe6062d591d8253bf9dbdc85db71c7e5176f0e74b5bef03d22bd/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/2b3483ee5d0ae371076d81f0f10e990c5290fbb9c5d9f19ef54c6c666afac886/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/85d16b042940088a27bf8323133e65625128bac15436fa458511ee3f0f4efead/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/98f2d01d9277e3d934a44a7c4a5f418e554e928a0d0d3c95ddeb964f2116fe5b/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/ce6a17e28622755322f5730f715b1914543055fd5cb24d53bb3e69929c446e14/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/8e32438c0ed601ee2400fe903b25f58fcbd3f33c2d427c20dfa0ca6704c7b32e/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/07a7078978ad53a7f76969584a8cbaae320ed579790fae569a56689085535357/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/84a88c74d91f63eeaa7c80c78e40734b3d3252818c34a051054feee9c79bf6f9/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/7be4ea859ab0a777ac299ee0b26c0d3ecce01f2f0a10f427106bec2bcad1594e/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/07f6c33bcacdd1db505eb900d3a71f44f4d7536adc79f7d5703d7940ecef6683/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/ec4534934ae463a1366e2748d6dd8aa6a70c9ec0a3141ac282bc41153493b2d8/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/74a2af0045c0885569f6025d744fdba7a983316715f01e243da89fb033f34aa2/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/fc22a9901336c494b709a6f48cdd21362449e7c188c7294fb949eb90f9dea262/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/450af76195c3c55a5353046849e4a6726c1923d5a270fd6602de2716b19c94c6/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/658e40c1b6b50d4a33bc1f771ef1bd4c9bfce2cbdd9cdc0e68d8d46e460d96ae/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/19b4422645f49887969307c7063a46247b3814433b6f4a83f97a2e5540d5a9b2/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/6a1b1f0af0f7e7bef411368c74f63e080035555e5fc1af06bf4e310abfb8bdf7/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/33a6303ed3ac258d154294d31ae3c0621a0fe8f015faf08c60c0b5787faf5b5f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b6755e6e6ceb974792a008615944fde35e2993c6cf7eaed6bd2cb321a5a4d15b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f43de89ad6ca282041cc5f81f06f61b08ba76d37c0e490cf1202e0b17c49d594/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7a22ee2e7a1582485b5f25bebbe1704fd54c8f817c0820736d76d9691c4bc69c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/cbb0d8bb2ad2e55516fa48d13ce99a370c468e7335d467af22312a1eb7857cdc/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/bc6d465c7183d1fb125b0acbd86c55c8a9aa13a6f4f49a4fd495f239cc39c0a5/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f9cf9b4edb58ea45cca9d897ce641465e498080c7d4dff88aef3e2d19d703adc/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/463bb6eb795177d371949af94d841a047fa4b762655b8e20ec68bf4c04848542/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 17:33:07.484674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.484292    5042 manager.go:217] Machine: {Timestamp:2026-11-02 17:33:07.483801788 +0000 UTC m=+0.063211166 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103198 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249ff89dfae4184363746ec6d8bda0 SystemUUID:ec249ff8-9dfa-e418-4363-746ec6d8bda0 BootID:d025974e-a6fc-49d4-82f5-779611ee8a04 Filesystems:[{Device:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~projected/kube-api-access-6lbkf DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5517a0556bcbb8c0d4b4edfd93724aa82c9782ef1e044c595e958e83ee135e4/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c58d6490-3f26-48cc-95eb-440097888b15/volumes/kubernetes.io~projected/kube-api-access-47sj4 DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a893c12e-173e-4d9c-8080-8fa9992d9396/volumes/kubernetes.io~projected/kube-api-access-6xszp DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~projected/kube-api-access-599dg DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/916899c2a6953307c553c8f0951d5f8509b53a819fd8a2c6f66f4b4c50b5567f/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1c250c2a7bcfd83b2af3098aff6c24b134e135a2632cdcef9daa9bff1b64299/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd0570a-e7e1-4681-937e-b357a6271f77/volumes/kubernetes.io~projected/kube-api-access-5m4kx DeviceMajor:0 DeviceMinor:349 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d55feb08-7024-4795-9a42-7345567fe78b/volumes/kubernetes.io~projected/kube-api-access-99htf DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84e9e4ccb9d0d6f7baa936417533b8e5f61be1b9aefdd9826eb5716c57657e66/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8645d924-656d-4790-a0d1-511c42b60e2b/volumes/kubernetes.io~projected/kube-api-access-rgfbs DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed398d4ed3fe62b2e99f95cdd3f0fa17f14e714400f955b6d16cd497abbc0be1/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17f8109d11cb244fc73d8f4cb5b142cffc8fa67743e88a3787c0ef4ca8b336d0/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b90ebafb-a030-43ce-a32a-edc04ce61c09/volumes/kubernetes.io~projected/kube-api-access-t42w6 DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17ccbff4-2818-4e70-b2f4-a64c92f789e8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e48a54-0431-4e67-a886-30ce38fc01ef/volumes/kubernetes.io~projected/kube-api-access-6m2n6 DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/032f9f33-d723-46ad-9d89-45956596eec5/volumes/kubernetes.io~projected/kube-api-access-q6tjt DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5dd0570a-e7e1-4681-937e-b357a6271f77/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:348 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52a1ad24-d5c4-49cf-aa71-b92f14918879/volumes/kubernetes.io~projected/kube-api-access-fwm2f DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98afd59c2779f69fd8200f96335b8b7a082fffd490d75f000065b79232baa5e9/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/volumes/kubernetes.io~projected/kube-api-access-xqcnt DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~projected/kube-api-access-p8dq8 DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1af870b5f6f7f1cf50c134fa4fd04d8877b89bf607dc56737d434d8b67241207/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce/volumes/kubernetes.io~projected/kube-api-access-mmqbq DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9510ad10d82aa0ce9db50f8f4306ab5a72fe0ccee7ebd1972bf0093ca9563cb2/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e1fc379-cd57-4b14-be52-b3975a5f6514/volumes/kubernetes.io~projected/kube-api-access-56zzf DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/814ca1bd-35f4-4b88-9d95-f9e1cc47845b/volumes/kubernetes.io~projected/kube-api-access-bczck DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5188d247-bf34-437c-bde7-0abc73c312f5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~projected/kube-api-access-ncq4r DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83d5b9bce013bf9da6256fa299397f84797ddbd4a54aff110f1a34cab8c28017/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volumes/kubernetes.io~projected/kube-api-access-89ld7 DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15fd7a8bb7eab3434816e074dd67990f6c83aba5f20cc85e5c4c71b5a607ee75/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72ce72d3-fe26-4612-ac57-078f9f7bbfbe/volumes/kubernetes.io~projected/kube-api-access-95m9q DeviceMajor:0 DeviceMinor:324 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/182b8e25bdd4e9bd12a78272f1ebad60247c370120874b43bee31a8c2e952e0a/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/505743f8-9737-44a8-987b-62f5d35212d3/volumes/kubernetes.io~projected/kube-api-access-87zcz DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52a1ad24-d5c4-49cf-aa71-b92f14918879/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1f693f58-c274-4b42-863d-7b4e73d8fb10/volumes/kubernetes.io~projected/kube-api-access-xjqh9 DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be9f38d9-38b2-42e3-8236-a3fa68f311db/volumes/kubernetes.io~projected/kube-api-access-jcm2m DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b497b468-2062-4cc3-9687-0994f99979fd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63048823-3df5-48fa-a65d-c1d789a319c9/volumes/kubernetes.io~projected/kube-api-access-wc8vv DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e759c85a4b26052307f7ac6277e6bb6ed7ebdb0b81d5c32dc6cdf08a5ab9f16/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99bb3840688efae0eb47f6664d15643c58bda8cd39653ded5e759427a4757f7c/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/269ca06c-b59b-496f-81aa-a08c525d0098/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/812e2a84-04de-4c0e-b8f1-f956381cb3fd/volumes/kubernetes.io~projected/kube-api-access-hwsfg DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7adb34aa-6a54-4a58-8761-49b5de183bba/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f559504e236d6245cd3b906c745e6346f8fee8755a3e063c2fe373059d75dc9e/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e/userdata/shm DeviceMajor:0 DeviceMinor:55 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85330f4e10db90c011e6f23705f938580a33d438278671a7a4f1ce361133638d/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e5bd7329e3ce6e54fb6fe787d3fbdf9da93d1b730f5a9bd445b83020f10d218/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f693f58-c274-4b42-863d-7b4e73d8fb10/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfc18b22507fc8d18f61e5b8ec9f425474409d1b75fd87d70ed18e5746cdc439/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d806b75d-88ca-4e51-94c3-cd45fd8888db/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e6d3493b4196c1151820997600e616794bf796bd05e31516e491dfe88cff0fb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80ff9264c004407516370cc33b4496c50ccf7b14e4a67d80c50a0aeeac30384a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd30ffd8-a9b2-46a3-b057-e25e98540942/volumes/kubernetes.io~projected/kube-api-access-rzhxr DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7b054e0f8085029622dbe61a0cbc89801a49cf04c15bf86423d93c423405668/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17ccbff4-2818-4e70-b2f4-a64c92f789e8/volumes/kubernetes.io~projected/kube-api-access-nq28j DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/565006a5-a1dd-4383-8078-f99d66892a75/volumes/kubernetes.io~projected/kube-api-access-vls44 DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b497b468-2062-4cc3-9687-0994f99979fd/volumes/kubernetes.io~projected/kube-api-access-6j4x8 DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64d7830f-395c-4694-a61c-c25f1c0adc5a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3c68c66-cc5c-4bc4-9937-691ece1b51b3/volumes/kubernetes.io~projected/kube-api-access-hxjlz DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8339155-7971-48b0-b048-babe5a75dc3e/volumes/kubernetes.io~projected/kube-api-access-k6ckt DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5742619-8be4-4c11-b8a3-abc18404e27b/volumes/kubernetes.io~projected/kube-api-access-czh9g DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33bd65d8c5af48f75b4e865db1cc4e591cb049145c0ef0baee2603f408034bdd/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d30a4f75582d476393a8e3f475665eda4ab2ad4411358e1e19f095dfad8cef80/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d61692dae0042c2a837497a57ce002852e58ffc1645b20462775f1dc249fd4ac/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:15fd7a8bb7eab34 MacAddress:8e:00:dd:60:77:a8 Speed:10000 Mtu:8901} {Name:1af870b5f6f7f1c MacAddress:0e:84:d1:86:b7:90 Speed:10000 Mtu:8901} {Name:33bd65d8c5af48f MacAddress:f6:74:89:71:ef:4d Speed:10000 Mtu:8901} {Name:85330f4e10db90c MacAddress:f2:67:12:7b:37:ff Speed:10000 Mtu:8901} {Name:8e759c85a4b2605 MacAddress:32:a4:7d:1a:fd:bd Speed:10000 Mtu:8901} {Name:916899c2a695330 MacAddress:7a:6e:06:06:7a:28 Speed:10000 Mtu:8901} {Name:9510ad10d82aa0c MacAddress:fa:d0:d5:45:d2:19 Speed:10000 Mtu:8901} {Name:98afd59c2779f69 MacAddress:1a:d0:48:4f:35:b0 Speed:10000 Mtu:8901} {Name:99bb3840688efae MacAddress:82:05:2f:d3:f1:00 Speed:10000 Mtu:8901} {Name:9e5bd7329e3ce6e MacAddress:2e:cc:59:e8:1b:23 Speed:10000 Mtu:8901} {Name:b1c250c2a7bcfd8 MacAddress:de:fb:a5:2e:59:06 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:a4:45:2b Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:f4:b8:d7:fd:3b Speed:0 Mtu:8901} {Name:d30a4f75582d476 MacAddress:82:55:9c:62:85:7f Speed:10000 Mtu:8901} {Name:dfc18b22507fc8d MacAddress:be:8e:5b:d7:2f:e5 Speed:10000 Mtu:8901} {Name:e5517a0556bcbb8 MacAddress:5a:03:6e:e9:be:9f Speed:10000 Mtu:8901} {Name:ed398d4ed3fe62b MacAddress:6e:5e:df:5d:1a:b5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:a4:45:2b Speed:0 Mtu:9001} {Name:f559504e236d624 MacAddress:12:17:5c:76:cd:73 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ea:c6:92:15:68:e7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:10:99:b6:b8:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:33:07.484924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.484679    5042 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:33:07.484924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.484729    5042 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:33:07.485126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485086    5042 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:33:07.485323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485130    5042 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-5.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:33:07.485372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485328    5042 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:33:07.485372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485346    5042 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:33:07.485372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485365    5042 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:33:07.485449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485392    5042 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:33:07.485449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485418    5042 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:33:07.485644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485632    5042 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:33:07.485781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485771    5042 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:33:07.485828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485819    5042 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:33:07.485883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485856    5042 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:33:07.485883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485872    5042 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:33:07.485975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485885    5042 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:33:07.485975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.485896    5042 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:33:07.486877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.486832    5042 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:33:07.486967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.486874    5042 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:33:07.487518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.487497    5042 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:33:07.489741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.489722    5042 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:33:07.489741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.489738    5042 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:33:07.489965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.489873    5042 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 17:33:07.490092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490077    5042 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:33:07.490144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490109    5042 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:33:07.490232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490222    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:33:07.490274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490238    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:33:07.490274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490245    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:33:07.490274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490252    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:33:07.490274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490261    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:33:07.490274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490268    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:33:07.490274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490274    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490279    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490286    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490292    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490300    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490310    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490344    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:33:07.490505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.490353    5042 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:33:07.494131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.494114    5042 server.go:1263] "Started kubelet"
Nov 02 17:33:07.494656 ip-10-0-135-5 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:33:07.495438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.494861    5042 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:33:07.495438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.494827    5042 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:33:07.495438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.494959    5042 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:33:07.495438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.495257    5042 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:33:07.498663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.498586    5042 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:33:07.502026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.502006    5042 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.503768    5042 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504007    5042 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:27:27 +0000 UTC" deadline="2028-08-07 02:13:55.536053574 +0000 UTC"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504033    5042 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15440h40m48.03202425s"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504205    5042 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504233    5042 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504243    5042 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504355    5042 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.504400    5042 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:33:07.504925 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.504431    5042 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-5.us-east-2.compute.internal\" not found"
Nov 02 17:33:07.507264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507190    5042 factory.go:55] Registering systemd factory
Nov 02 17:33:07.507264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507210    5042 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:33:07.507406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507322    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17ccbff4-2818-4e70-b2f4-a64c92f789e8" volumeName="kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-config" seLinuxMountContext=""
Nov 02 17:33:07.507406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507354    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="505743f8-9737-44a8-987b-62f5d35212d3" volumeName="kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-multus-daemon-config" seLinuxMountContext=""
Nov 02 17:33:07.507406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507368    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd0570a-e7e1-4681-937e-b357a6271f77" volumeName="kubernetes.io/secret/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-key" seLinuxMountContext=""
Nov 02 17:33:07.507406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507381    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="812e2a84-04de-4c0e-b8f1-f956381cb3fd" volumeName="kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg" seLinuxMountContext=""
Nov 02 17:33:07.507406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507394    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d806b75d-88ca-4e51-94c3-cd45fd8888db" volumeName="kubernetes.io/configmap/d806b75d-88ca-4e51-94c3-cd45fd8888db-konnectivity-ca" seLinuxMountContext=""
Nov 02 17:33:07.507406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507405    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3c68c66-cc5c-4bc4-9937-691ece1b51b3" volumeName="kubernetes.io/projected/f3c68c66-cc5c-4bc4-9937-691ece1b51b3-kube-api-access-hxjlz" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507417    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269ca06c-b59b-496f-81aa-a08c525d0098" volumeName="kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-config" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507429    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5188d247-bf34-437c-bde7-0abc73c312f5" volumeName="kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507443    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a1ad24-d5c4-49cf-aa71-b92f14918879" volumeName="kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-service-ca-bundle" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507456    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507469    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="814ca1bd-35f4-4b88-9d95-f9e1cc47845b" volumeName="kubernetes.io/projected/814ca1bd-35f4-4b88-9d95-f9e1cc47845b-kube-api-access-bczck" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507481    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd30ffd8-a9b2-46a3-b057-e25e98540942" volumeName="kubernetes.io/projected/fd30ffd8-a9b2-46a3-b057-e25e98540942-kube-api-access-rzhxr" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507495    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17ccbff4-2818-4e70-b2f4-a64c92f789e8" volumeName="kubernetes.io/secret/17ccbff4-2818-4e70-b2f4-a64c92f789e8-serving-cert" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507509    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8645d924-656d-4790-a0d1-511c42b60e2b" volumeName="kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507525    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507538    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63048823-3df5-48fa-a65d-c1d789a319c9" volumeName="kubernetes.io/empty-dir/63048823-3df5-48fa-a65d-c1d789a319c9-tmp-dir" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507551    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507563    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce" volumeName="kubernetes.io/configmap/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-iptables-alerter-script" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507575    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17ccbff4-2818-4e70-b2f4-a64c92f789e8" volumeName="kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-trusted-ca" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507586    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d7830f-395c-4694-a61c-c25f1c0adc5a" volumeName="kubernetes.io/projected/64d7830f-395c-4694-a61c-c25f1c0adc5a-kube-api-access-ncq4r" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507598    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ce72d3-fe26-4612-ac57-078f9f7bbfbe" volumeName="kubernetes.io/empty-dir/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-data-volume" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507609    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb34aa-6a54-4a58-8761-49b5de183bba" volumeName="kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-stats-auth" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507623    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507655    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e48a54-0431-4e67-a886-30ce38fc01ef" volumeName="kubernetes.io/configmap/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-config" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507668    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="032f9f33-d723-46ad-9d89-45956596eec5" volumeName="kubernetes.io/empty-dir/032f9f33-d723-46ad-9d89-45956596eec5-tmp-dir" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507681    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="505743f8-9737-44a8-987b-62f5d35212d3" volumeName="kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:33:07.507696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507704    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd0570a-e7e1-4681-937e-b357a6271f77" volumeName="kubernetes.io/configmap/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-cabundle" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507716    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb34aa-6a54-4a58-8761-49b5de183bba" volumeName="kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-default-certificate" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507735    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b497b468-2062-4cc3-9687-0994f99979fd" volumeName="kubernetes.io/configmap/b497b468-2062-4cc3-9687-0994f99979fd-config" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507747    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b497b468-2062-4cc3-9687-0994f99979fd" volumeName="kubernetes.io/secret/b497b468-2062-4cc3-9687-0994f99979fd-serving-cert" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507758    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be9f38d9-38b2-42e3-8236-a3fa68f311db" volumeName="kubernetes.io/projected/be9f38d9-38b2-42e3-8236-a3fa68f311db-kube-api-access-jcm2m" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507770    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="505743f8-9737-44a8-987b-62f5d35212d3" volumeName="kubernetes.io/projected/505743f8-9737-44a8-987b-62f5d35212d3-kube-api-access-87zcz" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507782    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565006a5-a1dd-4383-8078-f99d66892a75" volumeName="kubernetes.io/projected/565006a5-a1dd-4383-8078-f99d66892a75-kube-api-access-vls44" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507794    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507806    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c58d6490-3f26-48cc-95eb-440097888b15" volumeName="kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507819    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a1ad24-d5c4-49cf-aa71-b92f14918879" volumeName="kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-tmp" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507830    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a1ad24-d5c4-49cf-aa71-b92f14918879" volumeName="kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-snapshots" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507872    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a1ad24-d5c4-49cf-aa71-b92f14918879" volumeName="kubernetes.io/projected/52a1ad24-d5c4-49cf-aa71-b92f14918879-kube-api-access-fwm2f" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507885    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="565006a5-a1dd-4383-8078-f99d66892a75" volumeName="kubernetes.io/configmap/565006a5-a1dd-4383-8078-f99d66892a75-telemetry-config" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507898    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8645d924-656d-4790-a0d1-511c42b60e2b" volumeName="kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507914    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b90ebafb-a030-43ce-a32a-edc04ce61c09" volumeName="kubernetes.io/projected/b90ebafb-a030-43ce-a32a-edc04ce61c09-kube-api-access-t42w6" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507926    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c58d6490-3f26-48cc-95eb-440097888b15" volumeName="kubernetes.io/projected/c58d6490-3f26-48cc-95eb-440097888b15-kube-api-access-47sj4" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507938    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f693f58-c274-4b42-863d-7b4e73d8fb10" volumeName="kubernetes.io/projected/1f693f58-c274-4b42-863d-7b4e73d8fb10-kube-api-access-xjqh9" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507951    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c" volumeName="kubernetes.io/projected/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-kube-api-access-xqcnt" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507963    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c" volumeName="kubernetes.io/secret/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-serving-cert" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507975    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269ca06c-b59b-496f-81aa-a08c525d0098" volumeName="kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-env-overrides" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507987    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.507999    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e1fc379-cd57-4b14-be52-b3975a5f6514" volumeName="kubernetes.io/configmap/7e1fc379-cd57-4b14-be52-b3975a5f6514-serviceca" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508014    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d806b75d-88ca-4e51-94c3-cd45fd8888db" volumeName="kubernetes.io/secret/d806b75d-88ca-4e51-94c3-cd45fd8888db-agent-certs" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508027    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f693f58-c274-4b42-863d-7b4e73d8fb10" volumeName="kubernetes.io/secret/1f693f58-c274-4b42-863d-7b4e73d8fb10-cert" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508040    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269ca06c-b59b-496f-81aa-a08c525d0098" volumeName="kubernetes.io/projected/269ca06c-b59b-496f-81aa-a08c525d0098-kube-api-access-89ld7" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508053    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508059    5042 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508067    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d7830f-395c-4694-a61c-c25f1c0adc5a" volumeName="kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-tuned" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508080    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c58d6490-3f26-48cc-95eb-440097888b15" volumeName="kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508091    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d55feb08-7024-4795-9a42-7345567fe78b" volumeName="kubernetes.io/projected/d55feb08-7024-4795-9a42-7345567fe78b-kube-api-access-99htf" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508104    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269ca06c-b59b-496f-81aa-a08c525d0098" volumeName="kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508126    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72ce72d3-fe26-4612-ac57-078f9f7bbfbe" volumeName="kubernetes.io/projected/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-api-access-95m9q" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508139    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8645d924-656d-4790-a0d1-511c42b60e2b" volumeName="kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508151    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508163    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508175    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508187    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="269ca06c-b59b-496f-81aa-a08c525d0098" volumeName="kubernetes.io/secret/269ca06c-b59b-496f-81aa-a08c525d0098-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508213    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a1ad24-d5c4-49cf-aa71-b92f14918879" volumeName="kubernetes.io/secret/52a1ad24-d5c4-49cf-aa71-b92f14918879-serving-cert" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508226    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d7830f-395c-4694-a61c-c25f1c0adc5a" volumeName="kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-tmp" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508240    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7adb34aa-6a54-4a58-8761-49b5de183bba" volumeName="kubernetes.io/projected/7adb34aa-6a54-4a58-8761-49b5de183bba-kube-api-access-p8dq8" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508252    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508264    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c58d6490-3f26-48cc-95eb-440097888b15" volumeName="kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508278    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8339155-7971-48b0-b048-babe5a75dc3e" volumeName="kubernetes.io/projected/f8339155-7971-48b0-b048-babe5a75dc3e-kube-api-access-k6ckt" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508290    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52a1ad24-d5c4-49cf-aa71-b92f14918879" volumeName="kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508302    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63048823-3df5-48fa-a65d-c1d789a319c9" volumeName="kubernetes.io/configmap/63048823-3df5-48fa-a65d-c1d789a319c9-config-volume" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508314    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63048823-3df5-48fa-a65d-c1d789a319c9" volumeName="kubernetes.io/projected/63048823-3df5-48fa-a65d-c1d789a319c9-kube-api-access-wc8vv" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508326    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce" volumeName="kubernetes.io/projected/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-kube-api-access-mmqbq" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508338    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="032f9f33-d723-46ad-9d89-45956596eec5" volumeName="kubernetes.io/projected/032f9f33-d723-46ad-9d89-45956596eec5-kube-api-access-q6tjt" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508350    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc" volumeName="kubernetes.io/configmap/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-nginx-conf" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508362    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17ccbff4-2818-4e70-b2f4-a64c92f789e8" volumeName="kubernetes.io/projected/17ccbff4-2818-4e70-b2f4-a64c92f789e8-kube-api-access-nq28j" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508374    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8645d924-656d-4790-a0d1-511c42b60e2b" volumeName="kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508386    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c" volumeName="kubernetes.io/configmap/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-config" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508398    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508411    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578aad49-7f4d-4150-ab54-bb2b9f74940e" volumeName="kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508428    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e1fc379-cd57-4b14-be52-b3975a5f6514" volumeName="kubernetes.io/projected/7e1fc379-cd57-4b14-be52-b3975a5f6514-kube-api-access-56zzf" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508441    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5dd0570a-e7e1-4681-937e-b357a6271f77" volumeName="kubernetes.io/projected/5dd0570a-e7e1-4681-937e-b357a6271f77-kube-api-access-5m4kx" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508453    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a25f6b9-47c2-45c0-9333-239c7103e570" volumeName="kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508466    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5742619-8be4-4c11-b8a3-abc18404e27b" volumeName="kubernetes.io/projected/a5742619-8be4-4c11-b8a3-abc18404e27b-kube-api-access-czh9g" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508479    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a893c12e-173e-4d9c-8080-8fa9992d9396" volumeName="kubernetes.io/projected/a893c12e-173e-4d9c-8080-8fa9992d9396-kube-api-access-6xszp" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508491    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b497b468-2062-4cc3-9687-0994f99979fd" volumeName="kubernetes.io/projected/b497b468-2062-4cc3-9687-0994f99979fd-kube-api-access-6j4x8" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508505    5042 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e48a54-0431-4e67-a886-30ce38fc01ef" volumeName="kubernetes.io/projected/d4e48a54-0431-4e67-a886-30ce38fc01ef-kube-api-access-6m2n6" seLinuxMountContext=""
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508518    5042 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508526    5042 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508747    5042 factory.go:153] Registering CRI-O factory
Nov 02 17:33:07.508710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508763    5042 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:33:07.511507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508861    5042 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:33:07.511507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508888    5042 factory.go:103] Registering Raw factory
Nov 02 17:33:07.511507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.508904    5042 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:33:07.511507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.509292    5042 manager.go:319] Starting recovery of all containers
Nov 02 17:33:07.515590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.512959    5042 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 17:33:07.520607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.520582    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:33:07.521833 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.521813    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:33:07.523866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.523730    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:33:07.526451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.526432    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:33:07.532229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.532192    5042 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:33:07.534997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.534972    5042 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:33:07.534997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.535000    5042 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:33:07.535146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.535025    5042 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:33:07.535146 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.535071    5042 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 17:33:07.538881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.538668    5042 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:33:07.551479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.551425    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="a28331119236f050979c3695b9343f1075ce17d99e6815da5d31d17ba3477805" exitCode=0
Nov 02 17:33:07.551479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.551476    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="3420fd0e9762d0f1537de79905d1f6726ba3d257d8dfaec76ef2984d54a783e8" exitCode=0
Nov 02 17:33:07.551666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.551486    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="da4a681bf350a399c37498d4164e12877a476bce5a0c77acab916c6a0cefb5dc" exitCode=0
Nov 02 17:33:07.551666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.551495    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="524e9066f6637451dac0e90f0253f97d74ab49243a83f0aba60480c3a82de32b" exitCode=0
Nov 02 17:33:07.551666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.551504    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="b01f8bc1e20f42d679f3e51396e5f77850c7d6fa0749118d3c5ad67ee93733da" exitCode=0
Nov 02 17:33:07.551666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.551511    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58d6490-3f26-48cc-95eb-440097888b15" containerID="c8859f2dab6a8f85135522dedfaadd16cc8532abeeeb09b05dfb1643cae95c52" exitCode=0
Nov 02 17:33:07.557826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.557791    5042 generic.go:356] "Generic (PLEG): container finished" podID="22d0975639c634403c0017e572800bfa" containerID="d817de27ce2c3085a921c1cfcd4f736a48dbdcaa7438bf66962375871300c76e" exitCode=0
Nov 02 17:33:07.569687 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.569663    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:33:07.570078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.570051    5042 generic.go:356] "Generic (PLEG): container finished" podID="269ca06c-b59b-496f-81aa-a08c525d0098" containerID="73fe9091f868c36d1fc896ecbd71b0f754b58e012d934a895c98ae03ccbef32d" exitCode=1
Nov 02 17:33:07.591284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.591252    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:33:07.591420 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.591294    5042 generic.go:356] "Generic (PLEG): container finished" podID="17ccbff4-2818-4e70-b2f4-a64c92f789e8" containerID="caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4" exitCode=255
Nov 02 17:33:07.608741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.608672    5042 manager.go:324] Recovery completed
Nov 02 17:33:07.631660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631631    5042 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:33:07.631660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631650    5042 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:33:07.631892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631672    5042 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:33:07.631892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631862    5042 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 17:33:07.631892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631878    5042 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 17:33:07.632009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631898    5042 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 17:33:07.632009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631903    5042 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 17:33:07.632009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631909    5042 policy_none.go:50] "Start"
Nov 02 17:33:07.632009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631929    5042 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:33:07.632009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.631943    5042 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:33:07.636115 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.636092    5042 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 17:33:07.645240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.645219    5042 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 17:33:07.645240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.645243    5042 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 17:33:07.645372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.645250    5042 policy_none.go:44] "Start"
Nov 02 17:33:07.649993 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.649959    5042 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:33:07.652575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.652561    5042 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:33:07.653100 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.652589    5042 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:33:07.653165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653104    5042 server.go:87] "Starting device plugin registration server"
Nov 02 17:33:07.653397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653386    5042 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:33:07.653452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653398    5042 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:33:07.653546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653525    5042 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:33:07.653694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653646    5042 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:33:07.653694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653695    5042 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:33:07.653811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653702    5042 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:33:07.653873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.653758    5042 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:33:07.653682804Z","UUID":"cbfb1e40-1c00-4f6d-af74-c1333a300f98","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:33:07.655211 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.655173    5042 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:33:07.655297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.655228    5042 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:33:07.655297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.655257    5042 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:33:07.754433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.754392    5042 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:33:07.755915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.755894    5042 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 02 17:33:07.756029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.755930    5042 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 02 17:33:07.756029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.755941    5042 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-5.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 02 17:33:07.756029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.755988    5042 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.768800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.768777    5042 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.768901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.768867    5042 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.836236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.836160    5042 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"]
Nov 02 17:33:07.837102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837086    5042 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.837177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837098    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"4cabb9f1887dd6d87ae5b4fba788119d","Type":"ContainerStarted","Data":"eb6514e0689b546289175346c7c885fe5db651ba3faf64052fda2dabc20df2ac"}
Nov 02 17:33:07.837177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837140    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"4cabb9f1887dd6d87ae5b4fba788119d","Type":"ContainerStarted","Data":"7865aebd5c2633dad54772130697a49e096a44f67946e9d15d15f3d503afc10f"}
Nov 02 17:33:07.837177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837155    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"22d0975639c634403c0017e572800bfa","Type":"ContainerStarted","Data":"4f21cbfd88444e763d335be4cc60157b05b725f4ed582b3da1edc6cf096af565"}
Nov 02 17:33:07.837177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837167    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"22d0975639c634403c0017e572800bfa","Type":"ContainerDied","Data":"d817de27ce2c3085a921c1cfcd4f736a48dbdcaa7438bf66962375871300c76e"}
Nov 02 17:33:07.837177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837179    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal" event={"ID":"22d0975639c634403c0017e572800bfa","Type":"ContainerStarted","Data":"de8ea7652539f276ee21bcfb98b4a29d44e75751fd24aa53e6f7d6e8daec340e"}
Nov 02 17:33:07.837485 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.837359    5042 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.850654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.850630    5042 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:33:07.850747 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.850688    5042 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.854142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.854128    5042 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:33:07.854204 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:07.854169    5042 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.916050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.915955    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.916050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.916015    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:07.916050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:07.916041    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4cabb9f1887dd6d87ae5b4fba788119d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"4cabb9f1887dd6d87ae5b4fba788119d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.016687 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.016638    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4cabb9f1887dd6d87ae5b4fba788119d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"4cabb9f1887dd6d87ae5b4fba788119d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.016863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.016662    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4cabb9f1887dd6d87ae5b4fba788119d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"4cabb9f1887dd6d87ae5b4fba788119d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.016863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.016725    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.016863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.016742    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.016863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.016775    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.016863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.016794    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d0975639c634403c0017e572800bfa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" (UID: \"22d0975639c634403c0017e572800bfa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.486612 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.486576    5042 apiserver.go:52] "Watching apiserver"
Nov 02 17:33:08.498992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.498968    5042 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:33:08.499653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.499629    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dzlmp","openshift-network-operator/iptables-alerter-msvsr","kube-system/kas-connection-checker-685895c7c9-892gj","openshift-multus/network-metrics-daemon-bq652","kube-system/global-pull-secret-syncer-b26bx","kube-system/konnectivity-agent-fh7z4","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm","openshift-image-registry/image-registry-7fd6588566-jdf7q","openshift-ingress/router-default-c7d5754d7-v6znk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-m9pzk","openshift-insights/periodic-gathering-rx2qf-5jt9h","openshift-ovn-kubernetes/ovnkube-node-pswgg","openshift-image-registry/image-registry-7987d8c6c-fplrw","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l","openshift-multus/multus-7lh5w","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g","openshift-image-registry/node-ca-dv585","openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs","openshift-console-operator/console-operator-7cd56b8dbc-622zn","openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z","openshift-service-ca/service-ca-5fffc97bdf-s5jkp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2","openshift-ingress-canary/ingress-canary-gqgqg","openshift-network-console/networking-console-plugin-86c478bb88-47kbn","kube-system/kas-connection-checker-685895c7c9-nwfc8","openshift-dns/dns-default-gfg92","openshift-network-diagnostics/network-check-target-pp7tz","openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc","openshift-insights/insights-runtime-extractor-hffvc","openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp","openshift-insights/insights-operator-7c4986d449-hk6xf","openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm","openshift-multus/multus-additional-cni-plugins-vng2x","kube-system/kas-connection-checker-685895c7c9-bxzmw","kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"]
Nov 02 17:33:08.501134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.501116    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.502377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.502342    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.502915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.502895    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.503523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.503499    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.503701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.503682    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.504783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.504764    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:08.504895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.504883    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.505131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505113    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 17:33:08.505256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505239    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 17:33:08.505313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505280    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z5gr8\""
Nov 02 17:33:08.505380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505318    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.505447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505400    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l5rdl\""
Nov 02 17:33:08.505886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505867    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xgjc4\""
Nov 02 17:33:08.505946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.505920    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-76wwc\""
Nov 02 17:33:08.507832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.507808    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mfww7\""
Nov 02 17:33:08.508173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.508155    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 17:33:08.508242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.508191    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.508386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.508365    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 17:33:08.508497 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.508483    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 17:33:08.509345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509316    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.509438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509348    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.509514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509493    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 17:33:08.509669 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509654    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 17:33:08.509735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509679    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.509779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509745    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.509882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.509868    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 17:33:08.510114 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510086    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.510173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510132    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 17:33:08.510245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510133    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2cgvl\""
Nov 02 17:33:08.510366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510352    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.510446 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510429    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.510516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510473    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 17:33:08.510516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510479    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:33:08.510620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510525    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-s9lzk\""
Nov 02 17:33:08.510620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510096    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 17:33:08.510620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510551    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:33:08.510762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.510617    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.511717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.511696    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 17:33:08.511807 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.511747    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.511885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.511705    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 17:33:08.512369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512351    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 17:33:08.512451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512405    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.512451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512432    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nq4sl\""
Nov 02 17:33:08.512451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512436    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.512585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512465    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.512585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512474    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 17:33:08.512585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512520    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 17:33:08.512585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512574    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 17:33:08.512748 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512622    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.512748 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512675    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-v5pqt\""
Nov 02 17:33:08.512827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512811    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.512996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.512969    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.513222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513201    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 17:33:08.513363 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513340    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9bqkr\""
Nov 02 17:33:08.513458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513413    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zpng6\""
Nov 02 17:33:08.513458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513448    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.513566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513529    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 17:33:08.513901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513884    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:33:08.514033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.513968    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.514082 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.514051    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.514202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.514186    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 17:33:08.514343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.514326    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 17:33:08.514879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.514866    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:08.516071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.516041    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:33:08.516071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.516053    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:33:08.516196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.516132    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 17:33:08.516582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.516568    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.516875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.516859    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zg2g5\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517008    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517034    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517080    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517092    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517094    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517115    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jl4vs\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517094    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517145    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517175    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.517244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517215    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wjf7q\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517270    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkfrw\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517403    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517476    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517478    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-k78hz\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517536    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517580    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dtld9\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517594    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517582    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517644    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517661    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttfxd\""
Nov 02 17:33:08.517740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517700    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.518076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.517807    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jmxfr\""
Nov 02 17:33:08.518259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518243    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:33:08.518299 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518267    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.518299 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518286    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99htf\" (UniqueName: \"kubernetes.io/projected/d55feb08-7024-4795-9a42-7345567fe78b-kube-api-access-99htf\") pod \"volume-data-source-validator-58c9f5f46d-4559l\" (UID: \"d55feb08-7024-4795-9a42-7345567fe78b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:33:08.518382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518305    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.518382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518325    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcm2m\" (UniqueName: \"kubernetes.io/projected/be9f38d9-38b2-42e3-8236-a3fa68f311db-kube-api-access-jcm2m\") pod \"network-check-source-6f78d59f99-bbsgc\" (UID: \"be9f38d9-38b2-42e3-8236-a3fa68f311db\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:33:08.518382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518349    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.518534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518425    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52a1ad24-d5c4-49cf-aa71-b92f14918879-serving-cert\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.518534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518461    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bczck\" (UniqueName: \"kubernetes.io/projected/814ca1bd-35f4-4b88-9d95-f9e1cc47845b-kube-api-access-bczck\") pod \"kas-connection-checker-685895c7c9-892gj\" (UID: \"814ca1bd-35f4-4b88-9d95-f9e1cc47845b\") " pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:33:08.518534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518491    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vls44\" (UniqueName: \"kubernetes.io/projected/565006a5-a1dd-4383-8078-f99d66892a75-kube-api-access-vls44\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.518534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518507    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:33:08.518534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518518    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518549    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dq8\" (UniqueName: \"kubernetes.io/projected/7adb34aa-6a54-4a58-8761-49b5de183bba-kube-api-access-p8dq8\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518574    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b497b468-2062-4cc3-9687-0994f99979fd-serving-cert\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518600    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjlz\" (UniqueName: \"kubernetes.io/projected/f3c68c66-cc5c-4bc4-9937-691ece1b51b3-kube-api-access-hxjlz\") pod \"kas-connection-checker-685895c7c9-nwfc8\" (UID: \"f3c68c66-cc5c-4bc4-9937-691ece1b51b3\") " pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518615    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518626    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2n6\" (UniqueName: \"kubernetes.io/projected/d4e48a54-0431-4e67-a886-30ce38fc01ef-kube-api-access-6m2n6\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518656    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17ccbff4-2818-4e70-b2f4-a64c92f789e8-serving-cert\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518729    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518762    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518774    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sbqnj\""
Nov 02 17:33:08.518802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518785    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/52a1ad24-d5c4-49cf-aa71-b92f14918879-serving-cert\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.519277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518815    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b497b468-2062-4cc3-9687-0994f99979fd-serving-cert\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:33:08.519277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518792    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:33:08.519277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518793    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.519277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518902    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.519277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.518863    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq28j\" (UniqueName: \"kubernetes.io/projected/17ccbff4-2818-4e70-b2f4-a64c92f789e8-kube-api-access-nq28j\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.519277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.519197    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.519577 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.519530    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-default-certificate\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.519636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.519594    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17ccbff4-2818-4e70-b2f4-a64c92f789e8-serving-cert\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.519795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.519650    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-default-certificate\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.519896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.519879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-stats-auth\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.520103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520082    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:33:08.520201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520092    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:33:08.520259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520086    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.520306 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520277    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4x8\" (UniqueName: \"kubernetes.io/projected/b497b468-2062-4cc3-9687-0994f99979fd-kube-api-access-6j4x8\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:33:08.520370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520349    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-stats-auth\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.520457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520433    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqcnt\" (UniqueName: \"kubernetes.io/projected/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-kube-api-access-xqcnt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:33:08.520581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520565    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-snapshots\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.520737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520718    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-snapshots\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.520804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520766    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b497b468-2062-4cc3-9687-0994f99979fd-config\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:33:08.520890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520818    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:08.520890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520874    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xszp\" (UniqueName: \"kubernetes.io/projected/a893c12e-173e-4d9c-8080-8fa9992d9396-kube-api-access-6xszp\") pod \"kas-connection-checker-685895c7c9-bxzmw\" (UID: \"a893c12e-173e-4d9c-8080-8fa9992d9396\") " pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:33:08.520997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520917    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:33:08.520997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.520981    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/565006a5-a1dd-4383-8078-f99d66892a75-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.521096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521020    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:08.521096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521053    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.521262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521245    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-service-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.521359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521343    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dcw65\""
Nov 02 17:33:08.521410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521285    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwm2f\" (UniqueName: \"kubernetes.io/projected/52a1ad24-d5c4-49cf-aa71-b92f14918879-kube-api-access-fwm2f\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.521467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521405    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-config\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.521467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521441    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.521561 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521483    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-tmp\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.521561 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521497    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.521662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521503    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.521662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521636    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52a1ad24-d5c4-49cf-aa71-b92f14918879-tmp\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.521920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.521898    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.522154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522134    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52a1ad24-d5c4-49cf-aa71-b92f14918879-service-ca-bundle\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.522218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522184    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:33:08.522315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522287    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b497b468-2062-4cc3-9687-0994f99979fd-config\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:33:08.522384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522345    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:08.522878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522607    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-config\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.522878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522733    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:33:08.522878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.522778    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:33:08.523578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.523555    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/565006a5-a1dd-4383-8078-f99d66892a75-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.523704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.523684    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/17ccbff4-2818-4e70-b2f4-a64c92f789e8-trusted-ca\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.524149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.524134    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.525651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.525629    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-52q82\""
Nov 02 17:33:08.525748 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.525648    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xjkc9\""
Nov 02 17:33:08.527039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.526680    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjlz\" (UniqueName: \"kubernetes.io/projected/f3c68c66-cc5c-4bc4-9937-691ece1b51b3-kube-api-access-hxjlz\") pod \"kas-connection-checker-685895c7c9-nwfc8\" (UID: \"f3c68c66-cc5c-4bc4-9937-691ece1b51b3\") " pod="kube-system/kas-connection-checker-685895c7c9-nwfc8"
Nov 02 17:33:08.527039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.526929    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcm2m\" (UniqueName: \"kubernetes.io/projected/be9f38d9-38b2-42e3-8236-a3fa68f311db-kube-api-access-jcm2m\") pod \"network-check-source-6f78d59f99-bbsgc\" (UID: \"be9f38d9-38b2-42e3-8236-a3fa68f311db\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-bbsgc"
Nov 02 17:33:08.527039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.526948    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99htf\" (UniqueName: \"kubernetes.io/projected/d55feb08-7024-4795-9a42-7345567fe78b-kube-api-access-99htf\") pod \"volume-data-source-validator-58c9f5f46d-4559l\" (UID: \"d55feb08-7024-4795-9a42-7345567fe78b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-4559l"
Nov 02 17:33:08.527039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq28j\" (UniqueName: \"kubernetes.io/projected/17ccbff4-2818-4e70-b2f4-a64c92f789e8-kube-api-access-nq28j\") pod \"console-operator-7cd56b8dbc-622zn\" (UID: \"17ccbff4-2818-4e70-b2f4-a64c92f789e8\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:08.527039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527017    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dp4wq\""
Nov 02 17:33:08.527934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527055    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bczck\" (UniqueName: \"kubernetes.io/projected/814ca1bd-35f4-4b88-9d95-f9e1cc47845b-kube-api-access-bczck\") pod \"kas-connection-checker-685895c7c9-892gj\" (UID: \"814ca1bd-35f4-4b88-9d95-f9e1cc47845b\") " pod="kube-system/kas-connection-checker-685895c7c9-892gj"
Nov 02 17:33:08.527934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527433    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dq8\" (UniqueName: \"kubernetes.io/projected/7adb34aa-6a54-4a58-8761-49b5de183bba-kube-api-access-p8dq8\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.527934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527449    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqcnt\" (UniqueName: \"kubernetes.io/projected/246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c-kube-api-access-xqcnt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-75jlm\" (UID: \"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm"
Nov 02 17:33:08.527934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527478    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vls44\" (UniqueName: \"kubernetes.io/projected/565006a5-a1dd-4383-8078-f99d66892a75-kube-api-access-vls44\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.527934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.527629    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2n6\" (UniqueName: \"kubernetes.io/projected/d4e48a54-0431-4e67-a886-30ce38fc01ef-kube-api-access-6m2n6\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.529287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.529270    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.529783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.529758    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:33:08.529888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.529791    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:33:08.529888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.529821    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.529996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.529907    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9wn4g\""
Nov 02 17:33:08.529996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.529975    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:33:08.530060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.530000    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vxnnv\""
Nov 02 17:33:08.530205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.530192    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.530493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.530478    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.530534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.530501    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wb2z5\""
Nov 02 17:33:08.531593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.531573    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4x8\" (UniqueName: \"kubernetes.io/projected/b497b468-2062-4cc3-9687-0994f99979fd-kube-api-access-6j4x8\") pod \"service-ca-operator-544d749c8c-9svzs\" (UID: \"b497b468-2062-4cc3-9687-0994f99979fd\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs"
Nov 02 17:33:08.531662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.531633    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 17:33:08.531715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.531660    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 17:33:08.531759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.531711    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 17:33:08.531759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.531755    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9fwwt\""
Nov 02 17:33:08.531825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.531783    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 17:33:08.532364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.532348    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xszp\" (UniqueName: \"kubernetes.io/projected/a893c12e-173e-4d9c-8080-8fa9992d9396-kube-api-access-6xszp\") pod \"kas-connection-checker-685895c7c9-bxzmw\" (UID: \"a893c12e-173e-4d9c-8080-8fa9992d9396\") " pod="kube-system/kas-connection-checker-685895c7c9-bxzmw"
Nov 02 17:33:08.533087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.533071    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwm2f\" (UniqueName: \"kubernetes.io/projected/52a1ad24-d5c4-49cf-aa71-b92f14918879-kube-api-access-fwm2f\") pod \"insights-operator-7c4986d449-hk6xf\" (UID: \"52a1ad24-d5c4-49cf-aa71-b92f14918879\") " pod="openshift-insights/insights-operator-7c4986d449-hk6xf"
Nov 02 17:33:08.596450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.596427    5042 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.596653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.596518    5042 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.605938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.605912    5042 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:33:08.611318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.611297    5042 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:33:08.611442 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.611373    5042 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.611442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.611403    5042 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:33:08.611556 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.611445    5042 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-5.us-east-2.compute.internal"
Nov 02 17:33:08.622274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622249    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d806b75d-88ca-4e51-94c3-cd45fd8888db-agent-certs\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:33:08.622393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622278    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.622393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622298    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032f9f33-d723-46ad-9d89-45956596eec5-tmp-dir\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.622393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622315    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-tuned\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.622393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622337    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-kubelet\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.622393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622374    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-config\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.622393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622393    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622410    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622431    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-lib-modules\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622434    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-tuned\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622458    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622495    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e1fc379-cd57-4b14-be52-b3975a5f6514-host\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.622571    5042 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:33:08.622615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/a5742619-8be4-4c11-b8a3-abc18404e27b-kube-api-access-czh9g\") pod \"migrator-66df5d496b-hjh9z\" (UID: \"a5742619-8be4-4c11-b8a3-abc18404e27b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.622662    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.122639673 +0000 UTC m=+1.702049054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622432    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/032f9f33-d723-46ad-9d89-45956596eec5-tmp-dir\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622703    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zcz\" (UniqueName: \"kubernetes.io/projected/505743f8-9737-44a8-987b-62f5d35212d3-kube-api-access-87zcz\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622804    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-config\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622809    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622926    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d806b75d-88ca-4e51-94c3-cd45fd8888db-agent-certs\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622919    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:08.622962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.622963    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhxr\" (UniqueName: \"kubernetes.io/projected/fd30ffd8-a9b2-46a3-b057-e25e98540942-kube-api-access-rzhxr\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623011    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623107    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-node-log\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623168    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-bin\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623198    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032f9f33-d723-46ad-9d89-45956596eec5-hosts-file\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623250    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623272    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-etc-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.623297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623292    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63048823-3df5-48fa-a65d-c1d789a319c9-tmp-dir\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623310    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623346    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623397    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63048823-3df5-48fa-a65d-c1d789a319c9-tmp-dir\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623447    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623480    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-systemd\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623494    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-tmp\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623532    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-cni-binary-copy\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623548    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64d7830f-395c-4694-a61c-c25f1c0adc5a-tmp\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.623582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623561    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-ovn\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623629    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623680    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-sys\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623726    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-k8s-cni-cncf-io\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623796    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-cni-binary-copy\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623798    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95m9q\" (UniqueName: \"kubernetes.io/projected/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-api-access-95m9q\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623967    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-var-lib-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.624012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.623994    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-iptables-alerter-script\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624053    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zzf\" (UniqueName: \"kubernetes.io/projected/7e1fc379-cd57-4b14-be52-b3975a5f6514-kube-api-access-56zzf\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624074    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-crio-socket\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624097    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-bin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624118    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624135    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624156    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624247    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncq4r\" (UniqueName: \"kubernetes.io/projected/64d7830f-395c-4694-a61c-c25f1c0adc5a-kube-api-access-ncq4r\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.624255    5042 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624286    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-kubelet\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624289    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-iptables-alerter-script\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.624304    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.124291608 +0000 UTC m=+1.703700972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624324    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-kubelet-config\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.624375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624343    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-dbus\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624395    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624417    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624433    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-key\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624448    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-conf\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624478    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-multus\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624582    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d806b75d-88ca-4e51-94c3-cd45fd8888db-konnectivity-ca\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624614    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-system-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624621    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624631    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624645    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-key\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624657    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-data-volume\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624756    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-cnibin\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624799    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d806b75d-88ca-4e51-94c3-cd45fd8888db-konnectivity-ca\") pod \"konnectivity-agent-fh7z4\" (UID: \"d806b75d-88ca-4e51-94c3-cd45fd8888db\") " pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624794    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-data-volume\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624800    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-device-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624883    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624894    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:08.624943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624926    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sj4\" (UniqueName: \"kubernetes.io/projected/c58d6490-3f26-48cc-95eb-440097888b15-kube-api-access-47sj4\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624962    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-log-socket\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.625000    5042 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.624997    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625040    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-script-lib\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.625066    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.125049892 +0000 UTC m=+1.704459275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625088    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89ld7\" (UniqueName: \"kubernetes.io/projected/269ca06c-b59b-496f-81aa-a08c525d0098-kube-api-access-89ld7\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625093    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7adb34aa-6a54-4a58-8761-49b5de183bba-service-ca-bundle\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625117    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e1fc379-cd57-4b14-be52-b3975a5f6514-serviceca\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625140    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625156    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8vv\" (UniqueName: \"kubernetes.io/projected/63048823-3df5-48fa-a65d-c1d789a319c9-kube-api-access-wc8vv\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625174    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625189    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625215    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625236    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-ovnkube-script-lib\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625322    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625345    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7e1fc379-cd57-4b14-be52-b3975a5f6514-serviceca\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625364    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625386    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-var-lib-kubelet\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625388    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625402    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-kubernetes\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625389    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625435    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5dd0570a-e7e1-4681-937e-b357a6271f77-signing-cabundle\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625440    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625526    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625559    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625599    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625609    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5188d247-bf34-437c-bde7-0abc73c312f5-original-pull-secret\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.625647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625656    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625693    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.625720    5042 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625721    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625717    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.625750    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.125741129 +0000 UTC m=+1.705150492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625765    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-modprobe-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625802    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-run\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625832    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-hostroot\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625884    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/269ca06c-b59b-496f-81aa-a08c525d0098-ovn-node-metrics-cert\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625904    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-systemd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625931    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tjt\" (UniqueName: \"kubernetes.io/projected/032f9f33-d723-46ad-9d89-45956596eec5-kube-api-access-q6tjt\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625954    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqh9\" (UniqueName: \"kubernetes.io/projected/1f693f58-c274-4b42-863d-7b4e73d8fb10-kube-api-access-xjqh9\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.625976    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-binary-copy\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626008    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63048823-3df5-48fa-a65d-c1d789a319c9-config-volume\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626044    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-socket-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626108    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-host-slash\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626143    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626153    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/269ca06c-b59b-496f-81aa-a08c525d0098-ovn-node-metrics-cert\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626175    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626199    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-etc-kubernetes\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c58d6490-3f26-48cc-95eb-440097888b15-cni-binary-copy\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626426    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63048823-3df5-48fa-a65d-c1d789a319c9-config-volume\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626438    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f693f58-c274-4b42-863d-7b4e73d8fb10-cert\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626474    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ckt\" (UniqueName: \"kubernetes.io/projected/f8339155-7971-48b0-b048-babe5a75dc3e-kube-api-access-k6ckt\") pod \"migrator-66df5d496b-cp2bp\" (UID: \"f8339155-7971-48b0-b048-babe5a75dc3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626541    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626683    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f693f58-c274-4b42-863d-7b4e73d8fb10-cert\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626753    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626762    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626779    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42w6\" (UniqueName: \"kubernetes.io/projected/b90ebafb-a030-43ce-a32a-edc04ce61c09-kube-api-access-t42w6\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.626792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626809    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-systemd-units\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626876    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626889    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626905    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-socket-dir-parent\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626945    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-conf-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-multus-certs\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.626999    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-registration-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627019    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-sys-fs\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627056    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627075    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627091    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-cnibin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627106    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-netns\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627148    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-system-cni-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627262    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627295    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-netd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627419    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627582    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627629    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627661    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysconfig\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627695    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-host\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.627810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627751    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-multus-daemon-config\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627824    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-slash\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627905    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627941    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-netns\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.627982    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-env-overrides\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628057    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/505743f8-9737-44a8-987b-62f5d35212d3-multus-daemon-config\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628070    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628098    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/269ca06c-b59b-496f-81aa-a08c525d0098-env-overrides\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628110    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628141    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmqbq\" (UniqueName: \"kubernetes.io/projected/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-kube-api-access-mmqbq\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.628547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628170    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m4kx\" (UniqueName: \"kubernetes.io/projected/5dd0570a-e7e1-4681-937e-b357a6271f77-kube-api-access-5m4kx\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:08.628827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.628810    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.629471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.629457    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhxr\" (UniqueName: \"kubernetes.io/projected/fd30ffd8-a9b2-46a3-b057-e25e98540942-kube-api-access-rzhxr\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:08.629910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.629895    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh9g\" (UniqueName: \"kubernetes.io/projected/a5742619-8be4-4c11-b8a3-abc18404e27b-kube-api-access-czh9g\") pod \"migrator-66df5d496b-hjh9z\" (UID: \"a5742619-8be4-4c11-b8a3-abc18404e27b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hjh9z"
Nov 02 17:33:08.633418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.633393    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsfg\" (UniqueName: \"kubernetes.io/projected/812e2a84-04de-4c0e-b8f1-f956381cb3fd-kube-api-access-hwsfg\") pod \"network-check-target-pp7tz\" (UID: \"812e2a84-04de-4c0e-b8f1-f956381cb3fd\") " pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:33:08.633626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.633602    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zcz\" (UniqueName: \"kubernetes.io/projected/505743f8-9737-44a8-987b-62f5d35212d3-kube-api-access-87zcz\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.633782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.633682    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.634020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.634004    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95m9q\" (UniqueName: \"kubernetes.io/projected/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-api-access-95m9q\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.634314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.634295    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncq4r\" (UniqueName: \"kubernetes.io/projected/64d7830f-395c-4694-a61c-c25f1c0adc5a-kube-api-access-ncq4r\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.635600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.635576    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8vv\" (UniqueName: \"kubernetes.io/projected/63048823-3df5-48fa-a65d-c1d789a319c9-kube-api-access-wc8vv\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.636157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636127    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") pod \"periodic-gathering-rx2qf-5jt9h\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") " pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:33:08.636350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636327    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.636559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636540    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ckt\" (UniqueName: \"kubernetes.io/projected/f8339155-7971-48b0-b048-babe5a75dc3e-kube-api-access-k6ckt\") pod \"migrator-66df5d496b-cp2bp\" (UID: \"f8339155-7971-48b0-b048-babe5a75dc3e\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-cp2bp"
Nov 02 17:33:08.636659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636624    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tjt\" (UniqueName: \"kubernetes.io/projected/032f9f33-d723-46ad-9d89-45956596eec5-kube-api-access-q6tjt\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.636744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636656    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sj4\" (UniqueName: \"kubernetes.io/projected/c58d6490-3f26-48cc-95eb-440097888b15-kube-api-access-47sj4\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.636918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636789    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zzf\" (UniqueName: \"kubernetes.io/projected/7e1fc379-cd57-4b14-be52-b3975a5f6514-kube-api-access-56zzf\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.636918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.636804    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.637152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.637135    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89ld7\" (UniqueName: \"kubernetes.io/projected/269ca06c-b59b-496f-81aa-a08c525d0098-kube-api-access-89ld7\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.637211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.637187    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.637281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.637263    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqh9\" (UniqueName: \"kubernetes.io/projected/1f693f58-c274-4b42-863d-7b4e73d8fb10-kube-api-access-xjqh9\") pod \"ingress-canary-gqgqg\" (UID: \"1f693f58-c274-4b42-863d-7b4e73d8fb10\") " pod="openshift-ingress-canary/ingress-canary-gqgqg"
Nov 02 17:33:08.637342 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.637321    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m4kx\" (UniqueName: \"kubernetes.io/projected/5dd0570a-e7e1-4681-937e-b357a6271f77-kube-api-access-5m4kx\") pod \"service-ca-5fffc97bdf-s5jkp\" (UID: \"5dd0570a-e7e1-4681-937e-b357a6271f77\") " pod="openshift-service-ca/service-ca-5fffc97bdf-s5jkp"
Nov 02 17:33:08.637408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.637386    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmqbq\" (UniqueName: \"kubernetes.io/projected/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-kube-api-access-mmqbq\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.637619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.637605    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42w6\" (UniqueName: \"kubernetes.io/projected/b90ebafb-a030-43ce-a32a-edc04ce61c09-kube-api-access-t42w6\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.729319 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729283    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-modprobe-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.729319 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729322    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-run\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729351    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-hostroot\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729373    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-systemd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-socket-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729419    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-host-slash\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729448    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-run\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729477    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-modprobe-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729480    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729460    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce-host-slash\") pod \"iptables-alerter-msvsr\" (UID: \"d8e2dfb2-b24d-4af8-ad36-7c4532ee90ce\") " pod="openshift-network-operator/iptables-alerter-msvsr"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729448    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-hostroot\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729495    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-socket-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729457    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-systemd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.729578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729533    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729592    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-etc-kubernetes\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729637    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-systemd-units\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729655    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-etc-kubernetes\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729674    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-systemd-units\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729676    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-socket-dir-parent\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729717    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-conf-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729724    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-socket-dir-parent\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729738    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-multus-certs\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729755    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-registration-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729769    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-sys-fs\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729766    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-conf-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729788    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-cnibin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729822    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-multus-certs\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729830    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-registration-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729859    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-netns\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729886    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-cnibin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729890    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-netns\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729858    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-sys-fs\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729891    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-system-cni-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729924    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-system-cni-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729931    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-netd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729960    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729967    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-netd\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729980    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysconfig\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.729993    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-host\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730008    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-slash\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730029    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730054    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-netns\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730066    5042 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730070    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-host\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730082    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysconfig\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730085    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-netns\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730100    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-kubelet\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730056    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-slash\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730125    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.230105933 +0000 UTC m=+1.809515312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730128    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-kubelet\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730161    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730164    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.730171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730190    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-lib-modules\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730208    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730225    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e1fc379-cd57-4b14-be52-b3975a5f6514-host\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730243    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730173    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730260    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-d\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730270    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-node-log\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730311    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-node-log\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730316    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-lib-modules\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730338    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7e1fc379-cd57-4b14-be52-b3975a5f6514-host\") pod \"node-ca-dv585\" (UID: \"7e1fc379-cd57-4b14-be52-b3975a5f6514\") " pod="openshift-image-registry/node-ca-dv585"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730342    5042 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730344    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.230318838 +0000 UTC m=+1.809728219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730417    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-bin\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730435    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032f9f33-d723-46ad-9d89-45956596eec5-hosts-file\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730449    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730455    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-cni-bin\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.730486    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.230461875 +0000 UTC m=+1.809871240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : secret "metrics-daemon-secret" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730510    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-multus-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730511    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/032f9f33-d723-46ad-9d89-45956596eec5-hosts-file\") pod \"node-resolver-dzlmp\" (UID: \"032f9f33-d723-46ad-9d89-45956596eec5\") " pod="openshift-dns/node-resolver-dzlmp"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730512    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-etc-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730543    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-etc-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730555    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-systemd\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730572    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-ovn\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730587    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730611    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-systemd\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730626    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-sys\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730653    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-run-ovn\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730674    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-sys\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730745    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-k8s-cni-cncf-io\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730779    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-var-lib-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730806    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-crio-socket\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730826    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-run-k8s-cni-cncf-io\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730830    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-bin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730881    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-bin\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730882    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-crio-socket\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730855    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-var-lib-openvswitch\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730880    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730913    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-run-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730938    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-kubelet\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.730981    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-kubelet-config\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731004    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-dbus\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731032    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-conf\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731045    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-kubelet\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731064    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-kubelet-config\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731080    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-multus\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731086    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731103    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-system-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731130    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-cnibin\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731139    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-system-cni-dir\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731132    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/505743f8-9737-44a8-987b-62f5d35212d3-host-var-lib-cni-multus\") pod \"multus-7lh5w\" (UID: \"505743f8-9737-44a8-987b-62f5d35212d3\") " pod="openshift-multus/multus-7lh5w"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731088    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5188d247-bf34-437c-bde7-0abc73c312f5-dbus\") pod \"global-pull-secret-syncer-b26bx\" (UID: \"5188d247-bf34-437c-bde7-0abc73c312f5\") " pod="kube-system/global-pull-secret-syncer-b26bx"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731148    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-device-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731161    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c58d6490-3f26-48cc-95eb-440097888b15-cnibin\") pod \"multus-additional-cni-plugins-vng2x\" (UID: \"c58d6490-3f26-48cc-95eb-440097888b15\") " pod="openshift-multus/multus-additional-cni-plugins-vng2x"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731176    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-sysctl-conf\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731188    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-device-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731189    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-log-socket\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731208    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-log-socket\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731215    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731245    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731253    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/269ca06c-b59b-496f-81aa-a08c525d0098-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pswgg\" (UID: \"269ca06c-b59b-496f-81aa-a08c525d0098\") " pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731269    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731294    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-var-lib-kubelet\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731309    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-kubernetes\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731308    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-etc-selinux\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731325    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731359    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-var-lib-kubelet\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731360    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731391    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b90ebafb-a030-43ce-a32a-edc04ce61c09-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qsph2\" (UID: \"b90ebafb-a030-43ce-a32a-edc04ce61c09\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qsph2"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.731366    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64d7830f-395c-4694-a61c-c25f1c0adc5a-etc-kubernetes\") pod \"tuned-m9pzk\" (UID: \"64d7830f-395c-4694-a61c-c25f1c0adc5a\") " pod="openshift-cluster-node-tuning-operator/tuned-m9pzk"
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.731419    5042 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.731424    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.731436    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.731458    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.231445338 +0000 UTC m=+1.810854709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:33:08.731684 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:08.731475    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:33:09.231464192 +0000 UTC m=+1.810873590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:33:08.805370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:08.805280    5042 scope.go:122] "RemoveContainer" containerID="caaa5c15f7a00db32ad373bb38a5a07dd10420dd9a861df5c7bda72d9a951da4"
Nov 02 17:33:09.134907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.134793    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:09.135076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.134943    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:09.135076 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.134957    5042 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:33:09.135076 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135029    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.135013441 +0000 UTC m=+2.714422832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:33:09.135076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.135048    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.135086    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135097    5042 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135163    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs podName:7adb34aa-6a54-4a58-8761-49b5de183bba nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.135146302 +0000 UTC m=+2.714555669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs") pod "router-default-c7d5754d7-v6znk" (UID: "7adb34aa-6a54-4a58-8761-49b5de183bba") : secret "router-metrics-certs-default" not found
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135166    5042 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135202    5042 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135215    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.13520128 +0000 UTC m=+2.714610653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:09.135284 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.135232    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.135222658 +0000 UTC m=+2.714632050 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:33:09.236242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.236206    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:09.236410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.236252    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:09.236410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.236309    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:09.236410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.236339    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:09.236410 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236380    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:09.236410 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236398    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.236415    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236435    5042 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236455    5042 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236473    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.236454345 +0000 UTC m=+2.815863726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236490    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236504    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236507    5042 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236507    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.236489227 +0000 UTC m=+2.815898593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236534    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.236527362 +0000 UTC m=+2.815936725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236554    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.23653959 +0000 UTC m=+2.815948970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:33:09.236578 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:09.236573    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:10.236562981 +0000 UTC m=+2.815972347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : secret "metrics-daemon-secret" not found
Nov 02 17:33:09.452051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.452020    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzlmp_032f9f33-d723-46ad-9d89-45956596eec5/dns-node-resolver/0.log"
Nov 02 17:33:09.601873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.601828    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:33:09.602261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:09.601954    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" event={"ID":"17ccbff4-2818-4e70-b2f4-a64c92f789e8","Type":"ContainerStarted","Data":"00639af1e84fe1ea5a23ce4506394fff84db6e3b54b253e3dbb6ce9ceb57d747"}
Nov 02 17:33:10.052313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.052281    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dv585_7e1fc379-cd57-4b14-be52-b3975a5f6514/node-ca/0.log"
Nov 02 17:33:10.147142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.147100    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:10.147142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.147156    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.147194    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.147254    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.147289    5042 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.147359    5042 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.147364    5042 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.147363    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.147347761 +0000 UTC m=+4.726757138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.147428    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.147414625 +0000 UTC m=+4.726823989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:33:10.147465 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.147439    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert podName:0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.147433026 +0000 UTC m=+4.726842390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-47kbn" (UID: "0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc") : secret "networking-console-plugin-cert" not found
Nov 02 17:33:10.147740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.147580    5042 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:33:10.149884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.149867    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adb34aa-6a54-4a58-8761-49b5de183bba-metrics-certs\") pod \"router-default-c7d5754d7-v6znk\" (UID: \"7adb34aa-6a54-4a58-8761-49b5de183bba\") " pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:10.248310 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.248268    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:10.248310 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.248310    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248408    5042 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248418    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248438    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248454    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls podName:63048823-3df5-48fa-a65d-c1d789a319c9 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.248443314 +0000 UTC m=+4.827852678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls") pod "dns-default-gfg92" (UID: "63048823-3df5-48fa-a65d-c1d789a319c9") : secret "dns-default-metrics-tls" not found
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248485    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.24846786 +0000 UTC m=+4.827877241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.248508    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.248539    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:10.248586 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.248586    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248645    5042 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248692    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.24867852 +0000 UTC m=+4.828087884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248714    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248732    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248717    5042 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248780    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.248764621 +0000 UTC m=+4.828174001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:33:10.248961 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:10.248808    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:12.248795222 +0000 UTC m=+4.828204585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : secret "metrics-daemon-secret" not found
Nov 02 17:33:10.304028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.303941    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:10.482605 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.482562    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:10.487757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.487730    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn"
Nov 02 17:33:10.580757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.580668    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c7d5754d7-v6znk"]
Nov 02 17:33:10.585744 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:10.585701    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adb34aa_6a54_4a58_8761_49b5de183bba.slice/crio-bb3080b54ed247ae800c737c0deff320f949a1e58ec25632c9ff12a1fcce848c WatchSource:0}: Error finding container bb3080b54ed247ae800c737c0deff320f949a1e58ec25632c9ff12a1fcce848c: Status 404 returned error can't find the container with id bb3080b54ed247ae800c737c0deff320f949a1e58ec25632c9ff12a1fcce848c
Nov 02 17:33:10.607040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.607013    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c7d5754d7-v6znk" event={"ID":"7adb34aa-6a54-4a58-8761-49b5de183bba","Type":"ContainerStarted","Data":"bb3080b54ed247ae800c737c0deff320f949a1e58ec25632c9ff12a1fcce848c"}
Nov 02 17:33:10.851935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:10.851809    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gqgqg_1f693f58-c274-4b42-863d-7b4e73d8fb10/serve-healthcheck-canary/0.log"
Nov 02 17:33:11.039448 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:11.039418    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fh7z4"
Nov 02 17:33:11.052579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:11.052553    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-cp2bp_f8339155-7971-48b0-b048-babe5a75dc3e/migrator/0.log"
Nov 02 17:33:11.252571 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:11.252542    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-hjh9z_a5742619-8be4-4c11-b8a3-abc18404e27b/migrator/0.log"
Nov 02 17:33:11.452797 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:11.452762    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-75jlm_246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c/kube-storage-version-migrator-operator/0.log"
Nov 02 17:33:11.612434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:11.612349    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c7d5754d7-v6znk" event={"ID":"7adb34aa-6a54-4a58-8761-49b5de183bba","Type":"ContainerStarted","Data":"97077edd9793d7bae291c0c5c4f5731ee3bc42842e801e3b01f0a80a11e3ae2f"}
Nov 02 17:33:11.896666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:11.896590    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:33:12.166783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.166696    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:12.166783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.166743    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:12.167057 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.166861    5042 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:12.167057 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.166927    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls podName:565006a5-a1dd-4383-8078-f99d66892a75 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:16.166911565 +0000 UTC m=+8.746320960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-s9wxm" (UID: "565006a5-a1dd-4383-8078-f99d66892a75") : secret "cluster-monitoring-operator-tls" not found
Nov 02 17:33:12.167057 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.166921    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:12.167057 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.166992    5042 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 17:33:12.167057 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.167053    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls podName:d4e48a54-0431-4e67-a886-30ce38fc01ef nodeName:}" failed. No retries permitted until 2026-11-02 17:33:16.167036587 +0000 UTC m=+8.746445965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-rpx9g" (UID: "d4e48a54-0431-4e67-a886-30ce38fc01ef") : secret "samples-operator-tls" not found
Nov 02 17:33:12.169285 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.169263    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-47kbn\" (UID: \"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:12.268098 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.268063    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.268108    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.268148    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.268168    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.268194    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268219    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268240    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fd6588566-jdf7q: secret "image-registry-tls" not found
Nov 02 17:33:12.268294 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268255    5042 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268298    5042 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268318    5042 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268319    5042 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7987d8c6c-fplrw: secret "image-registry-tls" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268303    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls podName:578aad49-7f4d-4150-ab54-bb2b9f74940e nodeName:}" failed. No retries permitted until 2026-11-02 17:33:16.268283129 +0000 UTC m=+8.847692515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls") pod "image-registry-7fd6588566-jdf7q" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e") : secret "image-registry-tls" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268419    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls podName:72ce72d3-fe26-4612-ac57-078f9f7bbfbe nodeName:}" failed. No retries permitted until 2026-11-02 17:33:16.268404331 +0000 UTC m=+8.847813694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hffvc" (UID: "72ce72d3-fe26-4612-ac57-078f9f7bbfbe") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268434    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs podName:fd30ffd8-a9b2-46a3-b057-e25e98540942 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:16.268424257 +0000 UTC m=+8.847833622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs") pod "network-metrics-daemon-bq652" (UID: "fd30ffd8-a9b2-46a3-b057-e25e98540942") : secret "metrics-daemon-secret" not found
Nov 02 17:33:12.268652 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:12.268451    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls podName:9a25f6b9-47c2-45c0-9333-239c7103e570 nodeName:}" failed. No retries permitted until 2026-11-02 17:33:16.268441043 +0000 UTC m=+8.847850407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls") pod "image-registry-7987d8c6c-fplrw" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570") : secret "image-registry-tls" not found
Nov 02 17:33:12.270763 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.270739    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63048823-3df5-48fa-a65d-c1d789a319c9-metrics-tls\") pod \"dns-default-gfg92\" (UID: \"63048823-3df5-48fa-a65d-c1d789a319c9\") " pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:12.304823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.304787    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:12.307466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.307442    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:12.405959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.405924    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn"
Nov 02 17:33:12.420646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.420613    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:12.580331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.580288    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:12.581681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.581662    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c7d5754d7-v6znk"
Nov 02 17:33:12.680216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.680146    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-47kbn"]
Nov 02 17:33:12.683300 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:12.683272    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4a9cee_81c7_46ce_b7c2_9c273dd7e6fc.slice/crio-f4259bbe6072893ad0a49e6851eaf53afed7cb2ae8a22d5e241c8552aa0c8b10 WatchSource:0}: Error finding container f4259bbe6072893ad0a49e6851eaf53afed7cb2ae8a22d5e241c8552aa0c8b10: Status 404 returned error can't find the container with id f4259bbe6072893ad0a49e6851eaf53afed7cb2ae8a22d5e241c8552aa0c8b10
Nov 02 17:33:12.685619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.685601    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:33:12.704470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:12.704441    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gfg92"]
Nov 02 17:33:12.707441 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:12.707418    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63048823_3df5_48fa_a65d_c1d789a319c9.slice/crio-c1ab774c14b334ea87e2f58c3848d4554336a5d62bca93613259acf3926bb19f WatchSource:0}: Error finding container c1ab774c14b334ea87e2f58c3848d4554336a5d62bca93613259acf3926bb19f: Status 404 returned error can't find the container with id c1ab774c14b334ea87e2f58c3848d4554336a5d62bca93613259acf3926bb19f
Nov 02 17:33:13.622668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:13.622623    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn" event={"ID":"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc","Type":"ContainerStarted","Data":"f4259bbe6072893ad0a49e6851eaf53afed7cb2ae8a22d5e241c8552aa0c8b10"}
Nov 02 17:33:13.624243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:13.624210    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfg92" event={"ID":"63048823-3df5-48fa-a65d-c1d789a319c9","Type":"ContainerStarted","Data":"c1ab774c14b334ea87e2f58c3848d4554336a5d62bca93613259acf3926bb19f"}
Nov 02 17:33:14.630179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:14.630087    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfg92" event={"ID":"63048823-3df5-48fa-a65d-c1d789a319c9","Type":"ContainerStarted","Data":"5c753f86746596dbd75f850898fb8c94b5bf4a9cd52b370ed25459650c5db02e"}
Nov 02 17:33:14.630179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:14.630144    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gfg92" event={"ID":"63048823-3df5-48fa-a65d-c1d789a319c9","Type":"ContainerStarted","Data":"e08427e86aea953e5f20bdfbf9521ee29636c7d160bc8795bc793856c2b81622"}
Nov 02 17:33:14.630179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:14.630176    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:14.631628 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:14.631604    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-47kbn" event={"ID":"0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc","Type":"ContainerStarted","Data":"09b6d5fe855f2e88424c2220b417c2e871a096dce3c7caece6dc2f9f2ad34fdb"}
Nov 02 17:33:14.721170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:14.721138    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:16.167513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.167482    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:16.167916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.167735    5042 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:33:16.209863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.209799    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:16.210046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.209897    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:16.212397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.212370    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/565006a5-a1dd-4383-8078-f99d66892a75-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-s9wxm\" (UID: \"565006a5-a1dd-4383-8078-f99d66892a75\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:16.212492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.212428    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d4e48a54-0431-4e67-a886-30ce38fc01ef-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-rpx9g\" (UID: \"d4e48a54-0431-4e67-a886-30ce38fc01ef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:16.302199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.302160    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"
Nov 02 17:33:16.303230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.303206    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"
Nov 02 17:33:16.311358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.311332    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:16.311499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.311468    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:16.311953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.311664    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:16.311953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.311713    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:16.314449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.314425    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"image-registry-7fd6588566-jdf7q\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") " pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:16.314700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.314530    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd30ffd8-a9b2-46a3-b057-e25e98540942-metrics-certs\") pod \"network-metrics-daemon-bq652\" (UID: \"fd30ffd8-a9b2-46a3-b057-e25e98540942\") " pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:16.314700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.314665    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"image-registry-7987d8c6c-fplrw\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") " pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:16.315170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.315150    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72ce72d3-fe26-4612-ac57-078f9f7bbfbe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hffvc\" (UID: \"72ce72d3-fe26-4612-ac57-078f9f7bbfbe\") " pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:16.315266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.315226    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bq652"
Nov 02 17:33:16.317229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.317209    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:16.325477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.325455    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hffvc"
Nov 02 17:33:16.603423 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.603392    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm"]
Nov 02 17:33:16.605782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.605756    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:16.606491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.606468    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g"]
Nov 02 17:33:16.606585 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:16.606548    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565006a5_a1dd_4383_8078_f99d66892a75.slice/crio-f7707d0e6e2083ba8b53734a0daaaccc2a6e8c64f366f80678cb31efc403a7ff WatchSource:0}: Error finding container f7707d0e6e2083ba8b53734a0daaaccc2a6e8c64f366f80678cb31efc403a7ff: Status 404 returned error can't find the container with id f7707d0e6e2083ba8b53734a0daaaccc2a6e8c64f366f80678cb31efc403a7ff
Nov 02 17:33:16.636102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.636020    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hffvc"]
Nov 02 17:33:16.639566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.639542    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bq652"]
Nov 02 17:33:16.639688 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.639610    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fd6588566-jdf7q"]
Nov 02 17:33:16.646418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.646385    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm" event={"ID":"565006a5-a1dd-4383-8078-f99d66892a75","Type":"ContainerStarted","Data":"f7707d0e6e2083ba8b53734a0daaaccc2a6e8c64f366f80678cb31efc403a7ff"}
Nov 02 17:33:16.649817 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:16.649789    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578aad49_7f4d_4150_ab54_bb2b9f74940e.slice/crio-512e55d8b5cc8bdb81f1eafee9d7a12ecfb6c19d2047a6d10b6fc176378a3227 WatchSource:0}: Error finding container 512e55d8b5cc8bdb81f1eafee9d7a12ecfb6c19d2047a6d10b6fc176378a3227: Status 404 returned error can't find the container with id 512e55d8b5cc8bdb81f1eafee9d7a12ecfb6c19d2047a6d10b6fc176378a3227
Nov 02 17:33:16.890816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:16.890783    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7987d8c6c-fplrw"]
Nov 02 17:33:16.894283 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:16.894254    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a25f6b9_47c2_45c0_9333_239c7103e570.slice/crio-2a1fa9946d7adc13ca3760c0284832087b41ec5220298d84baded54f352e3a92 WatchSource:0}: Error finding container 2a1fa9946d7adc13ca3760c0284832087b41ec5220298d84baded54f352e3a92: Status 404 returned error can't find the container with id 2a1fa9946d7adc13ca3760c0284832087b41ec5220298d84baded54f352e3a92
Nov 02 17:33:17.654961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.654919    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw" event={"ID":"9a25f6b9-47c2-45c0-9333-239c7103e570","Type":"ContainerStarted","Data":"6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e"}
Nov 02 17:33:17.655425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.654978    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw" event={"ID":"9a25f6b9-47c2-45c0-9333-239c7103e570","Type":"ContainerStarted","Data":"2a1fa9946d7adc13ca3760c0284832087b41ec5220298d84baded54f352e3a92"}
Nov 02 17:33:17.655425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.655124    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:17.658231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.658199    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q" event={"ID":"578aad49-7f4d-4150-ab54-bb2b9f74940e","Type":"ContainerStarted","Data":"21a8cc9c8b66455ceaad64da72c614c9cc9da1ae509cdec2bc5a07f9bdbe27e7"}
Nov 02 17:33:17.658378 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.658256    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q" event={"ID":"578aad49-7f4d-4150-ab54-bb2b9f74940e","Type":"ContainerStarted","Data":"512e55d8b5cc8bdb81f1eafee9d7a12ecfb6c19d2047a6d10b6fc176378a3227"}
Nov 02 17:33:17.661474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.661444    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bq652" event={"ID":"fd30ffd8-a9b2-46a3-b057-e25e98540942","Type":"ContainerStarted","Data":"2bf9f3f5a8ead0cf54b615e263f7d4f4eba3a9bbf5c0821892a6dae377a80306"}
Nov 02 17:33:17.663994 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.663968    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hffvc" event={"ID":"72ce72d3-fe26-4612-ac57-078f9f7bbfbe","Type":"ContainerStarted","Data":"d96317237e1a0511b339e2a0ddacb21a616d685ea1789af262b21ffe85da0e1c"}
Nov 02 17:33:17.664119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.664026    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hffvc" event={"ID":"72ce72d3-fe26-4612-ac57-078f9f7bbfbe","Type":"ContainerStarted","Data":"1699815953ece45ed9117f7a787b9ecf19ff25ce352f5cacb11b67973b59bc05"}
Nov 02 17:33:17.664119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.664053    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hffvc" event={"ID":"72ce72d3-fe26-4612-ac57-078f9f7bbfbe","Type":"ContainerStarted","Data":"fff603947e28341db8d9ca834df053741ab346c7a6d57bd159e31831b025cbd8"}
Nov 02 17:33:17.665905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:17.665810    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g" event={"ID":"d4e48a54-0431-4e67-a886-30ce38fc01ef","Type":"ContainerStarted","Data":"4070e2f32e71c57bfe8395129ac38f183a39921ff0a927f428840ca1376bc596"}
Nov 02 17:33:18.673061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:18.673020    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bq652" event={"ID":"fd30ffd8-a9b2-46a3-b057-e25e98540942","Type":"ContainerStarted","Data":"17556b72adf8c907bc285891f7233fbbd6a55b64284de944dc7b0c8154071070"}
Nov 02 17:33:18.673555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:18.673089    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bq652" event={"ID":"fd30ffd8-a9b2-46a3-b057-e25e98540942","Type":"ContainerStarted","Data":"79366b1619f2ab6c391121c98c7d97fc79df589929bc126c1f83e99d3ce4072e"}
Nov 02 17:33:18.673555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:18.673216    5042 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:33:19.172061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:19.172024    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:19.678423 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:19.678321    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-s9wxm" event={"ID":"565006a5-a1dd-4383-8078-f99d66892a75","Type":"ContainerStarted","Data":"edccc1f7e3671a22b71308eb07e407133da9687da5e8ce8ad7db06dae26a441f"}
Nov 02 17:33:19.680681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:19.680656    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hffvc" event={"ID":"72ce72d3-fe26-4612-ac57-078f9f7bbfbe","Type":"ContainerStarted","Data":"9d38421ab63e77555b89296a56c0834db1d8705968cd37aa466479c15264a17c"}
Nov 02 17:33:19.682504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:19.682480    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g" event={"ID":"d4e48a54-0431-4e67-a886-30ce38fc01ef","Type":"ContainerStarted","Data":"52ac89a15b5a3f3caa2d45057f2819b9cd66d183b43e3679ff52bbd3a20e27c3"}
Nov 02 17:33:19.682625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:19.682529    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-rpx9g" event={"ID":"d4e48a54-0431-4e67-a886-30ce38fc01ef","Type":"ContainerStarted","Data":"416ad4d49041783315f71f41c3979ba5735f0e361b10ff9250cc6bf872fb2784"}
Nov 02 17:33:20.251751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:20.251715    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pswgg"
Nov 02 17:33:24.349968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.349936    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"]
Nov 02 17:33:24.354981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.354958    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:24.367998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.367960    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 17:33:24.367998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.367974    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-95rhg\""
Nov 02 17:33:24.369834 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.369807    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"]
Nov 02 17:33:24.370681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.370656    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7987d8c6c-fplrw"]
Nov 02 17:33:24.447440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.447406    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-98bw2"]
Nov 02 17:33:24.451102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.451076    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.456662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.456634    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8cc4cf56c-hnwrt"]
Nov 02 17:33:24.460230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.460205    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bb6c9c46c-66dr9"]
Nov 02 17:33:24.460353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.460327    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.462942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.462914    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 17:33:24.467031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.465424    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.467031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.465576    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-vfcbz\""
Nov 02 17:33:24.474045 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.474020    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 17:33:24.474652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.474614    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-98bw2"]
Nov 02 17:33:24.481909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.481884    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb6c9c46c-66dr9"]
Nov 02 17:33:24.482041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.481930    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 17:33:24.482041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.481947    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 17:33:24.482041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.481987    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 17:33:24.490719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.490692    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fa18f4d4-d9fa-4802-b90b-e7044d222b39-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vktlz\" (UID: \"fa18f4d4-d9fa-4802-b90b-e7044d222b39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:24.499339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.499313    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8cc4cf56c-hnwrt"]
Nov 02 17:33:24.564951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.564915    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 17:33:24.565117 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.565006    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 17:33:24.565117 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.565013    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4n2v\""
Nov 02 17:33:24.592104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592069    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7skf\" (UniqueName: \"kubernetes.io/projected/6377c122-4ad5-4829-a2df-ed251a70d3bc-kube-api-access-j7skf\") pod \"downloads-6dd789c8b7-98bw2\" (UID: \"6377c122-4ad5-4829-a2df-ed251a70d3bc\") " pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.592104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592105    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.592316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592138    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fa18f4d4-d9fa-4802-b90b-e7044d222b39-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vktlz\" (UID: \"fa18f4d4-d9fa-4802-b90b-e7044d222b39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:24.592316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592255    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-registry-tls\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592364    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/417a9298-1e3c-46bb-965c-aab3c75b8728-ca-trust-extracted\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592425    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.592473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592449    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.592507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592489    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.592545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592535    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/417a9298-1e3c-46bb-965c-aab3c75b8728-installation-pull-secrets\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592585    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.592640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592609    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g4kl\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-kube-api-access-9g4kl\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592632    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6377c122-4ad5-4829-a2df-ed251a70d3bc-tmp\") pod \"downloads-6dd789c8b7-98bw2\" (UID: \"6377c122-4ad5-4829-a2df-ed251a70d3bc\") " pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.592730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592658    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/417a9298-1e3c-46bb-965c-aab3c75b8728-registry-certificates\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592698    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.592831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592804    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/417a9298-1e3c-46bb-965c-aab3c75b8728-trusted-ca\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592870    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/417a9298-1e3c-46bb-965c-aab3c75b8728-image-registry-private-configuration\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592888    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-bound-sa-token\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.592979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.592906    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9ht\" (UniqueName: \"kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.594938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.594922    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fa18f4d4-d9fa-4802-b90b-e7044d222b39-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-vktlz\" (UID: \"fa18f4d4-d9fa-4802-b90b-e7044d222b39\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:24.637748 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.637669    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gfg92"
Nov 02 17:33:24.669915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.669886    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:24.693523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693484    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/417a9298-1e3c-46bb-965c-aab3c75b8728-image-registry-private-configuration\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.693523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693518    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-bound-sa-token\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.693853 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693815    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9ht\" (UniqueName: \"kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.693937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693888    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7skf\" (UniqueName: \"kubernetes.io/projected/6377c122-4ad5-4829-a2df-ed251a70d3bc-kube-api-access-j7skf\") pod \"downloads-6dd789c8b7-98bw2\" (UID: \"6377c122-4ad5-4829-a2df-ed251a70d3bc\") " pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.693937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693905    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.694037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693938    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-registry-tls\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.694037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693954    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/417a9298-1e3c-46bb-965c-aab3c75b8728-ca-trust-extracted\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.694037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693980    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.694037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.693998    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694043    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694069    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/417a9298-1e3c-46bb-965c-aab3c75b8728-installation-pull-secrets\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694098    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694116    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g4kl\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-kube-api-access-9g4kl\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694138    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6377c122-4ad5-4829-a2df-ed251a70d3bc-tmp\") pod \"downloads-6dd789c8b7-98bw2\" (UID: \"6377c122-4ad5-4829-a2df-ed251a70d3bc\") " pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694158    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/417a9298-1e3c-46bb-965c-aab3c75b8728-registry-certificates\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.694232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694175    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.695003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.694827    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.695113 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.695040    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/417a9298-1e3c-46bb-965c-aab3c75b8728-trusted-ca\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.696266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.696237    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.697597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.696570    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/417a9298-1e3c-46bb-965c-aab3c75b8728-trusted-ca\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.697597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.696918    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/417a9298-1e3c-46bb-965c-aab3c75b8728-ca-trust-extracted\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.697597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.697246    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6377c122-4ad5-4829-a2df-ed251a70d3bc-tmp\") pod \"downloads-6dd789c8b7-98bw2\" (UID: \"6377c122-4ad5-4829-a2df-ed251a70d3bc\") " pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.697597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.697442    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/417a9298-1e3c-46bb-965c-aab3c75b8728-image-registry-private-configuration\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.697597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.697553    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.697597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.697565    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.697888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.697793    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.697927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.697893    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/417a9298-1e3c-46bb-965c-aab3c75b8728-registry-certificates\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.698026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.698010    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.699635 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.699606    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-registry-tls\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.701726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.700405    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/417a9298-1e3c-46bb-965c-aab3c75b8728-installation-pull-secrets\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.704227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.704206    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-bound-sa-token\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.704473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.704452    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7skf\" (UniqueName: \"kubernetes.io/projected/6377c122-4ad5-4829-a2df-ed251a70d3bc-kube-api-access-j7skf\") pod \"downloads-6dd789c8b7-98bw2\" (UID: \"6377c122-4ad5-4829-a2df-ed251a70d3bc\") " pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.705107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.705081    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9ht\" (UniqueName: \"kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht\") pod \"console-8cc4cf56c-hnwrt\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") " pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.707467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.707448    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g4kl\" (UniqueName: \"kubernetes.io/projected/417a9298-1e3c-46bb-965c-aab3c75b8728-kube-api-access-9g4kl\") pod \"image-registry-5bb6c9c46c-66dr9\" (UID: \"417a9298-1e3c-46bb-965c-aab3c75b8728\") " pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.765318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.765282    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:24.778238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.778031    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:24.784324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.784303    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:24.952233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:24.952207    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"]
Nov 02 17:33:24.954676 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:24.954648    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa18f4d4_d9fa_4802_b90b_e7044d222b39.slice/crio-671f83bf4b6521dae7d62f9e9b7321dbca904fd5872b0b260287de8487697dd1 WatchSource:0}: Error finding container 671f83bf4b6521dae7d62f9e9b7321dbca904fd5872b0b260287de8487697dd1: Status 404 returned error can't find the container with id 671f83bf4b6521dae7d62f9e9b7321dbca904fd5872b0b260287de8487697dd1
Nov 02 17:33:25.053817 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.053786    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-98bw2"]
Nov 02 17:33:25.056980 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:25.056954    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6377c122_4ad5_4829_a2df_ed251a70d3bc.slice/crio-8634838a26c0ed1c4ad9a30ae6448c90944fc326c6b141710a0765dc4d1ebee2 WatchSource:0}: Error finding container 8634838a26c0ed1c4ad9a30ae6448c90944fc326c6b141710a0765dc4d1ebee2: Status 404 returned error can't find the container with id 8634838a26c0ed1c4ad9a30ae6448c90944fc326c6b141710a0765dc4d1ebee2
Nov 02 17:33:25.073713 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.073688    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8cc4cf56c-hnwrt"]
Nov 02 17:33:25.076494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.076472    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bb6c9c46c-66dr9"]
Nov 02 17:33:25.076923 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:25.076898    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9692736c_0929_420f_8e8d_7e35565d5cb0.slice/crio-9cea8a105ef2956ef74be84c971d2749ba13c8e56879fa2fc3678fc4203b06bd WatchSource:0}: Error finding container 9cea8a105ef2956ef74be84c971d2749ba13c8e56879fa2fc3678fc4203b06bd: Status 404 returned error can't find the container with id 9cea8a105ef2956ef74be84c971d2749ba13c8e56879fa2fc3678fc4203b06bd
Nov 02 17:33:25.080046 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:25.080025    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417a9298_1e3c_46bb_965c_aab3c75b8728.slice/crio-077372ede55f59e39bdc21f733e213504000176c08a0cfed72f322c60db068cf WatchSource:0}: Error finding container 077372ede55f59e39bdc21f733e213504000176c08a0cfed72f322c60db068cf: Status 404 returned error can't find the container with id 077372ede55f59e39bdc21f733e213504000176c08a0cfed72f322c60db068cf
Nov 02 17:33:25.713464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.713375    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8cc4cf56c-hnwrt" event={"ID":"9692736c-0929-420f-8e8d-7e35565d5cb0","Type":"ContainerStarted","Data":"9cea8a105ef2956ef74be84c971d2749ba13c8e56879fa2fc3678fc4203b06bd"}
Nov 02 17:33:25.715476 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.715450    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-98bw2" event={"ID":"6377c122-4ad5-4829-a2df-ed251a70d3bc","Type":"ContainerStarted","Data":"8634838a26c0ed1c4ad9a30ae6448c90944fc326c6b141710a0765dc4d1ebee2"}
Nov 02 17:33:25.718198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.717305    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9" event={"ID":"417a9298-1e3c-46bb-965c-aab3c75b8728","Type":"ContainerStarted","Data":"8a8bcabc744821bf0349d14a30c5d0ee8484119d9041c5b8774f8255a259a5fe"}
Nov 02 17:33:25.718198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.717352    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9" event={"ID":"417a9298-1e3c-46bb-965c-aab3c75b8728","Type":"ContainerStarted","Data":"077372ede55f59e39bdc21f733e213504000176c08a0cfed72f322c60db068cf"}
Nov 02 17:33:25.718198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.718153    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:25.721372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.721337    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz" event={"ID":"fa18f4d4-d9fa-4802-b90b-e7044d222b39","Type":"ContainerStarted","Data":"671f83bf4b6521dae7d62f9e9b7321dbca904fd5872b0b260287de8487697dd1"}
Nov 02 17:33:25.739556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:25.738985    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9" podStartSLOduration=1.738965262 podStartE2EDuration="1.738965262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:25.737882632 +0000 UTC m=+18.317292021" watchObservedRunningTime="2026-11-02 17:33:25.738965262 +0000 UTC m=+18.318374649"
Nov 02 17:33:26.728473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:26.728425    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz" event={"ID":"fa18f4d4-d9fa-4802-b90b-e7044d222b39","Type":"ContainerStarted","Data":"7f6830a0d9956f243747c65ceb382093adb8f95450736b4bc1620b71d2827219"}
Nov 02 17:33:26.728940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:26.728618    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:26.735060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:26.735033    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz"
Nov 02 17:33:26.761879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:26.761804    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz" podStartSLOduration=1.616548663 podStartE2EDuration="2.76178876s" totalImagesPullingTime="1.145240097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:26.745105529 +0000 UTC m=+19.324514926" watchObservedRunningTime="2026-11-02 17:33:26.76178876 +0000 UTC m=+19.341198144"
Nov 02 17:33:28.740805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:28.740763    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8cc4cf56c-hnwrt" event={"ID":"9692736c-0929-420f-8e8d-7e35565d5cb0","Type":"ContainerStarted","Data":"06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec"}
Nov 02 17:33:28.763801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:28.763745    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8cc4cf56c-hnwrt" podStartSLOduration=1.591851686 podStartE2EDuration="4.763729178s" totalImagesPullingTime="3.171877492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:28.762272885 +0000 UTC m=+21.341682270" watchObservedRunningTime="2026-11-02 17:33:28.763729178 +0000 UTC m=+21.343138564"
Nov 02 17:33:31.923722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.923678    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f68nl"]
Nov 02 17:33:31.954196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.954166    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968238    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnfnv\" (UniqueName: \"kubernetes.io/projected/470d6f34-0ce2-446f-b498-e411eb263932-kube-api-access-hnfnv\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968337    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-textfile\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968374    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/470d6f34-0ce2-446f-b498-e411eb263932-metrics-client-ca\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968402    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-tls\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968592 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968504    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-wtmp\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968592 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968538    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-sys\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968605    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-accelerators-collector-config\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968673    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-root\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.968699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.968696    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:31.989502 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.989414    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 17:33:31.989752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.989725    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 17:33:31.989752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.989736    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 17:33:31.989960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.989804    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6p4gz\""
Nov 02 17:33:31.989960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:31.989888    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 17:33:32.069549 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069508    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-accelerators-collector-config\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.069738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069561    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-root\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.069738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069592    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.069738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069644    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnfnv\" (UniqueName: \"kubernetes.io/projected/470d6f34-0ce2-446f-b498-e411eb263932-kube-api-access-hnfnv\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.069738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069708    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-textfile\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.069738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069711    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-root\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069750    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/470d6f34-0ce2-446f-b498-e411eb263932-metrics-client-ca\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069777    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-tls\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069826    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-wtmp\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069878    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-sys\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.069954    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-sys\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.070116    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-accelerators-collector-config\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.070256    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-textfile\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.070393    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-wtmp\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.070790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.070768    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/470d6f34-0ce2-446f-b498-e411eb263932-metrics-client-ca\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.072925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.072899    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-tls\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.072925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.072923    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/470d6f34-0ce2-446f-b498-e411eb263932-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.078456 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.078432    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnfnv\" (UniqueName: \"kubernetes.io/projected/470d6f34-0ce2-446f-b498-e411eb263932-kube-api-access-hnfnv\") pod \"node-exporter-f68nl\" (UID: \"470d6f34-0ce2-446f-b498-e411eb263932\") " pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.326376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.326343    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f68nl"
Nov 02 17:33:32.365607 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:33:32.365573    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470d6f34_0ce2_446f_b498_e411eb263932.slice/crio-dfa13497a76e61b1135315e40380a7d37a273fb98938d5ee179879700734c003 WatchSource:0}: Error finding container dfa13497a76e61b1135315e40380a7d37a273fb98938d5ee179879700734c003: Status 404 returned error can't find the container with id dfa13497a76e61b1135315e40380a7d37a273fb98938d5ee179879700734c003
Nov 02 17:33:32.761256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:32.761224    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f68nl" event={"ID":"470d6f34-0ce2-446f-b498-e411eb263932","Type":"ContainerStarted","Data":"dfa13497a76e61b1135315e40380a7d37a273fb98938d5ee179879700734c003"}
Nov 02 17:33:34.376630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:34.376603    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:34.774342 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:34.774302    5042 generic.go:356] "Generic (PLEG): container finished" podID="470d6f34-0ce2-446f-b498-e411eb263932" containerID="231fcb8345d484d9833b9e3b569013f793d14c06a0416573c0746ecf77e2abe5" exitCode=0
Nov 02 17:33:34.774533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:34.774392    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f68nl" event={"ID":"470d6f34-0ce2-446f-b498-e411eb263932","Type":"ContainerDied","Data":"231fcb8345d484d9833b9e3b569013f793d14c06a0416573c0746ecf77e2abe5"}
Nov 02 17:33:34.780137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:34.778813    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:34.780137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:34.778871    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:34.784931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:34.784904    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:35.783260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:35.783216    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f68nl" event={"ID":"470d6f34-0ce2-446f-b498-e411eb263932","Type":"ContainerStarted","Data":"d15f9c3df6e9ad4a13688c83f01be1168b99961184cf9975ad52655b42fd0661"}
Nov 02 17:33:35.783731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:35.783290    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f68nl" event={"ID":"470d6f34-0ce2-446f-b498-e411eb263932","Type":"ContainerStarted","Data":"4a6783cdb25d0b2643d036b0599fc76639d553cacc55786f10aff03ec6e4d95d"}
Nov 02 17:33:35.787458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:35.787432    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:33:35.804968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:35.804869    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f68nl" podStartSLOduration=3.151919719 podStartE2EDuration="4.804828589s" totalImagesPullingTime="1.547924553s" totalInitContainerRuntime="104.984317ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:35.803403608 +0000 UTC m=+28.382812994" watchObservedRunningTime="2026-11-02 17:33:35.804828589 +0000 UTC m=+28.384237977"
Nov 02 17:33:39.316391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:39.316348    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8cc4cf56c-hnwrt"]
Nov 02 17:33:39.687696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:39.687656    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:33:41.815275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:41.815198    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-98bw2" event={"ID":"6377c122-4ad5-4829-a2df-ed251a70d3bc","Type":"ContainerStarted","Data":"ba996509a3b27c1c753f133813a3f9a9c0b493569b1a9b5371b93ffbf2bae52b"}
Nov 02 17:33:41.816326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:41.816303    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:41.836640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:41.836587    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-98bw2" podStartSLOduration=1.354062006 podStartE2EDuration="17.836569403s" totalImagesPullingTime="16.482507397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:41.83510463 +0000 UTC m=+34.414514017" watchObservedRunningTime="2026-11-02 17:33:41.836569403 +0000 UTC m=+34.415978789"
Nov 02 17:33:41.840001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:41.839973    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-98bw2"
Nov 02 17:33:41.900364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:41.900330    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pp7tz"
Nov 02 17:33:47.739439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:47.739349    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9"
Nov 02 17:33:47.800118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:47.800072    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd6588566-jdf7q"]
Nov 02 17:33:48.544876 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:48.544816    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:33:49.396276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.396202    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw" podUID="9a25f6b9-47c2-45c0-9333-239c7103e570" containerName="registry" containerID="cri-o://6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e" gracePeriod=30
Nov 02 17:33:49.726079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.726047    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:49.850071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850038    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850085    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850140    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850190    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850232    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850295    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850346    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850383    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") pod \"9a25f6b9-47c2-45c0-9333-239c7103e570\" (UID: \"9a25f6b9-47c2-45c0-9333-239c7103e570\") "
Nov 02 17:33:49.850646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850507    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:33:49.850646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850605    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:33:49.850742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.850662    5042 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-certificates\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.854214 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.854150    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "kube-api-access-599dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:33:49.856393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.856253    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:33:49.856815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.856419    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:33:49.856815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.856494    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:33:49.858534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.858507    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:33:49.859580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.859550    5042 generic.go:356] "Generic (PLEG): container finished" podID="9a25f6b9-47c2-45c0-9333-239c7103e570" containerID="6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e" exitCode=0
Nov 02 17:33:49.859915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.859610    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw"
Nov 02 17:33:49.859915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.859641    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw" event={"ID":"9a25f6b9-47c2-45c0-9333-239c7103e570","Type":"ContainerDied","Data":"6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e"}
Nov 02 17:33:49.859915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.859701    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7987d8c6c-fplrw" event={"ID":"9a25f6b9-47c2-45c0-9333-239c7103e570","Type":"ContainerDied","Data":"2a1fa9946d7adc13ca3760c0284832087b41ec5220298d84baded54f352e3a92"}
Nov 02 17:33:49.859915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.859731    5042 scope.go:122] "RemoveContainer" containerID="6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e"
Nov 02 17:33:49.861004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.860917    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted" pod "9a25f6b9-47c2-45c0-9333-239c7103e570" (UID: "9a25f6b9-47c2-45c0-9333-239c7103e570"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:33:49.877774 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.877744    5042 scope.go:122] "RemoveContainer" containerID="6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e"
Nov 02 17:33:49.878161 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:33:49.878134    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e\": container with ID starting with 6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e not found: ID does not exist" containerID="6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e"
Nov 02 17:33:49.878250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.878172    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e"} err="failed to get container status \"6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e\": rpc error: code = NotFound desc = could not find container \"6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e\": container with ID starting with 6ecb6afd9a8148c5b6ab73913ddc22654733781b0b8890dd86e91fa44f7fe06e not found: ID does not exist"
Nov 02 17:33:49.951987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.951947    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599dg\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-kube-api-access-599dg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.951987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.951984    5042 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-bound-sa-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.951987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.951997    5042 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a25f6b9-47c2-45c0-9333-239c7103e570-registry-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.952247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.952010    5042 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a25f6b9-47c2-45c0-9333-239c7103e570-trusted-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.952247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.952023    5042 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a25f6b9-47c2-45c0-9333-239c7103e570-ca-trust-extracted\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.952247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.952036    5042 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-image-registry-private-configuration\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:49.952247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:49.952048    5042 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a25f6b9-47c2-45c0-9333-239c7103e570-installation-pull-secrets\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:33:50.186273 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:50.186233    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7987d8c6c-fplrw"]
Nov 02 17:33:50.191050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:50.191021    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7987d8c6c-fplrw"]
Nov 02 17:33:51.541159 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:33:51.541126    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a25f6b9-47c2-45c0-9333-239c7103e570" path="/var/lib/kubelet/pods/9a25f6b9-47c2-45c0-9333-239c7103e570/volumes"
Nov 02 17:34:04.344155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.344094    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8cc4cf56c-hnwrt" podUID="9692736c-0929-420f-8e8d-7e35565d5cb0" containerName="console" containerID="cri-o://06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec" gracePeriod=15
Nov 02 17:34:04.592190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.592158    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:34:04.684343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684268    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht\" (UniqueName: \"kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684324    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684368    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684406    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684441    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684478    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684506    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert\") pod \"9692736c-0929-420f-8e8d-7e35565d5cb0\" (UID: \"9692736c-0929-420f-8e8d-7e35565d5cb0\") "
Nov 02 17:34:04.684944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684792    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:34:04.685008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684971    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:04.685008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.684967    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:04.685411 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.685387    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:04.686909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.686884    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "kube-api-access-fn9ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:04.687148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.687128    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:04.687236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.687219    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert" pod "9692736c-0929-420f-8e8d-7e35565d5cb0" (UID: "9692736c-0929-420f-8e8d-7e35565d5cb0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:04.785133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785095    5042 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-console-config\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.785133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785129    5042 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-oauth-config\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.785133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785141    5042 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.785422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785153    5042 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9692736c-0929-420f-8e8d-7e35565d5cb0-tmp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.785422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785165    5042 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9692736c-0929-420f-8e8d-7e35565d5cb0-console-serving-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.785422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785176    5042 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9692736c-0929-420f-8e8d-7e35565d5cb0-oauth-serving-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.785422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.785190    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9ht\" (UniqueName: \"kubernetes.io/projected/9692736c-0929-420f-8e8d-7e35565d5cb0-kube-api-access-fn9ht\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:04.929173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.929141    5042 generic.go:356] "Generic (PLEG): container finished" podID="9692736c-0929-420f-8e8d-7e35565d5cb0" containerID="06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec" exitCode=0
Nov 02 17:34:04.929322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.929197    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8cc4cf56c-hnwrt"
Nov 02 17:34:04.929322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.929198    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8cc4cf56c-hnwrt" event={"ID":"9692736c-0929-420f-8e8d-7e35565d5cb0","Type":"ContainerDied","Data":"06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec"}
Nov 02 17:34:04.929434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.929333    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8cc4cf56c-hnwrt" event={"ID":"9692736c-0929-420f-8e8d-7e35565d5cb0","Type":"ContainerDied","Data":"9cea8a105ef2956ef74be84c971d2749ba13c8e56879fa2fc3678fc4203b06bd"}
Nov 02 17:34:04.929434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.929353    5042 scope.go:122] "RemoveContainer" containerID="06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec"
Nov 02 17:34:04.938753 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.938727    5042 scope.go:122] "RemoveContainer" containerID="06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec"
Nov 02 17:34:04.939074 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:34:04.939054    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec\": container with ID starting with 06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec not found: ID does not exist" containerID="06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec"
Nov 02 17:34:04.939132 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.939083    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec"} err="failed to get container status \"06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec\": rpc error: code = NotFound desc = could not find container \"06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec\": container with ID starting with 06f286eaa9e40f22f5f5903c9e71517892afc7b83795d91426cc0ca278985dec not found: ID does not exist"
Nov 02 17:34:04.950416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.950388    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8cc4cf56c-hnwrt"]
Nov 02 17:34:04.958406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:04.958383    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8cc4cf56c-hnwrt"]
Nov 02 17:34:05.540925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:05.540885    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9692736c-0929-420f-8e8d-7e35565d5cb0" path="/var/lib/kubelet/pods/9692736c-0929-420f-8e8d-7e35565d5cb0/volumes"
Nov 02 17:34:12.829037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:12.828969    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q" podUID="578aad49-7f4d-4150-ab54-bb2b9f74940e" containerName="registry" containerID="cri-o://21a8cc9c8b66455ceaad64da72c614c9cc9da1ae509cdec2bc5a07f9bdbe27e7" gracePeriod=30
Nov 02 17:34:12.968085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:12.968049    5042 generic.go:356] "Generic (PLEG): container finished" podID="b497b468-2062-4cc3-9687-0994f99979fd" containerID="4d5cccac9be678913fe2c10a3137fc35a5b2cd49811693d5939ba9df2086baa8" exitCode=0
Nov 02 17:34:12.968255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:12.968077    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs" event={"ID":"b497b468-2062-4cc3-9687-0994f99979fd","Type":"ContainerDied","Data":"4d5cccac9be678913fe2c10a3137fc35a5b2cd49811693d5939ba9df2086baa8"}
Nov 02 17:34:12.968541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:12.968521    5042 scope.go:122] "RemoveContainer" containerID="4d5cccac9be678913fe2c10a3137fc35a5b2cd49811693d5939ba9df2086baa8"
Nov 02 17:34:12.971590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:12.971564    5042 generic.go:356] "Generic (PLEG): container finished" podID="578aad49-7f4d-4150-ab54-bb2b9f74940e" containerID="21a8cc9c8b66455ceaad64da72c614c9cc9da1ae509cdec2bc5a07f9bdbe27e7" exitCode=0
Nov 02 17:34:12.971943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:12.971597    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q" event={"ID":"578aad49-7f4d-4150-ab54-bb2b9f74940e","Type":"ContainerDied","Data":"21a8cc9c8b66455ceaad64da72c614c9cc9da1ae509cdec2bc5a07f9bdbe27e7"}
Nov 02 17:34:13.110379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.110358    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:34:13.155377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155346    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155408    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155463    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155492    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155560    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155601    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155639    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155673    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") pod \"578aad49-7f4d-4150-ab54-bb2b9f74940e\" (UID: \"578aad49-7f4d-4150-ab54-bb2b9f74940e\") "
Nov 02 17:34:13.155981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.155953    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:13.156801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.156651    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:13.158566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.158541    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:13.158692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.158573    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "kube-api-access-6lbkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:13.158692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.158641    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:13.158692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.158662    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:13.158799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.158707    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:34:13.163110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.163085    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted" pod "578aad49-7f4d-4150-ab54-bb2b9f74940e" (UID: "578aad49-7f4d-4150-ab54-bb2b9f74940e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:34:13.257043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.256965    5042 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/578aad49-7f4d-4150-ab54-bb2b9f74940e-ca-trust-extracted\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.256995    5042 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.257007    5042 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-image-registry-private-configuration\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.257018    5042 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-registry-certificates\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.257035    5042 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-bound-sa-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.257044    5042 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/578aad49-7f4d-4150-ab54-bb2b9f74940e-installation-pull-secrets\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.257054    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lbkf\" (UniqueName: \"kubernetes.io/projected/578aad49-7f4d-4150-ab54-bb2b9f74940e-kube-api-access-6lbkf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.257354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.257063    5042 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/578aad49-7f4d-4150-ab54-bb2b9f74940e-trusted-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:13.976855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.976810    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q"
Nov 02 17:34:13.977322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.976859    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fd6588566-jdf7q" event={"ID":"578aad49-7f4d-4150-ab54-bb2b9f74940e","Type":"ContainerDied","Data":"512e55d8b5cc8bdb81f1eafee9d7a12ecfb6c19d2047a6d10b6fc176378a3227"}
Nov 02 17:34:13.977322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.976927    5042 scope.go:122] "RemoveContainer" containerID="21a8cc9c8b66455ceaad64da72c614c9cc9da1ae509cdec2bc5a07f9bdbe27e7"
Nov 02 17:34:13.978916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:13.978891    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-9svzs" event={"ID":"b497b468-2062-4cc3-9687-0994f99979fd","Type":"ContainerStarted","Data":"bd0c9e59d89f80ee82b59cb5792194dcf55932ae632cad962b82926bcd934d02"}
Nov 02 17:34:14.013278 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:14.013245    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fd6588566-jdf7q"]
Nov 02 17:34:14.018207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:14.018182    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fd6588566-jdf7q"]
Nov 02 17:34:15.541147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:15.541116    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="578aad49-7f4d-4150-ab54-bb2b9f74940e" path="/var/lib/kubelet/pods/578aad49-7f4d-4150-ab54-bb2b9f74940e/volumes"
Nov 02 17:34:18.537351 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:34:18.537316    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:34:27.031640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:27.031603    5042 generic.go:356] "Generic (PLEG): container finished" podID="8645d924-656d-4790-a0d1-511c42b60e2b" containerID="a000ce97a577eec15b230643f05b74f723441a9f4b77cac41053ebdb41c45d6b" exitCode=0
Nov 02 17:34:27.032035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:27.031673    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h" event={"ID":"8645d924-656d-4790-a0d1-511c42b60e2b","Type":"ContainerDied","Data":"a000ce97a577eec15b230643f05b74f723441a9f4b77cac41053ebdb41c45d6b"}
Nov 02 17:34:28.190725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.190688    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:34:28.290021 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.289924    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") pod \"8645d924-656d-4790-a0d1-511c42b60e2b\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") "
Nov 02 17:34:28.290021 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.290004    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") pod \"8645d924-656d-4790-a0d1-511c42b60e2b\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") "
Nov 02 17:34:28.290245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.290029    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") pod \"8645d924-656d-4790-a0d1-511c42b60e2b\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") "
Nov 02 17:34:28.290245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.290105    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") pod \"8645d924-656d-4790-a0d1-511c42b60e2b\" (UID: \"8645d924-656d-4790-a0d1-511c42b60e2b\") "
Nov 02 17:34:28.290351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.290309    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle" pod "8645d924-656d-4790-a0d1-511c42b60e2b" (UID: "8645d924-656d-4790-a0d1-511c42b60e2b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:28.290406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.290380    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path" pod "8645d924-656d-4790-a0d1-511c42b60e2b" (UID: "8645d924-656d-4790-a0d1-511c42b60e2b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:34:28.290566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.290541    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle" pod "8645d924-656d-4790-a0d1-511c42b60e2b" (UID: "8645d924-656d-4790-a0d1-511c42b60e2b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:28.292632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.292610    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs" pod "8645d924-656d-4790-a0d1-511c42b60e2b" (UID: "8645d924-656d-4790-a0d1-511c42b60e2b"). InnerVolumeSpecName "kube-api-access-rgfbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:28.390782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.390745    5042 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-trusted-ca-bundle\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:28.390782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.390775    5042 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8645d924-656d-4790-a0d1-511c42b60e2b-service-ca-bundle\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:28.390782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.390785    5042 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/8645d924-656d-4790-a0d1-511c42b60e2b-archives-path\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:28.391048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:28.390794    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgfbs\" (UniqueName: \"kubernetes.io/projected/8645d924-656d-4790-a0d1-511c42b60e2b-kube-api-access-rgfbs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:34:29.041024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:29.040998    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h"
Nov 02 17:34:29.041024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:29.041019    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rx2qf-5jt9h" event={"ID":"8645d924-656d-4790-a0d1-511c42b60e2b","Type":"ContainerDied","Data":"98afd59c2779f69fd8200f96335b8b7a082fffd490d75f000065b79232baa5e9"}
Nov 02 17:34:29.041236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:29.041060    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98afd59c2779f69fd8200f96335b8b7a082fffd490d75f000065b79232baa5e9"
Nov 02 17:34:38.077545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.077510    5042 generic.go:356] "Generic (PLEG): container finished" podID="52a1ad24-d5c4-49cf-aa71-b92f14918879" containerID="eedb64b5917afc4e6302fa181807d401fcda8e9c3efc59d4771f0f461922f154" exitCode=0
Nov 02 17:34:38.077992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.077556    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hk6xf" event={"ID":"52a1ad24-d5c4-49cf-aa71-b92f14918879","Type":"ContainerDied","Data":"eedb64b5917afc4e6302fa181807d401fcda8e9c3efc59d4771f0f461922f154"}
Nov 02 17:34:38.077992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.077894    5042 scope.go:122] "RemoveContainer" containerID="eedb64b5917afc4e6302fa181807d401fcda8e9c3efc59d4771f0f461922f154"
Nov 02 17:34:38.503434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503394    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fjcwd-h6mhp"]
Nov 02 17:34:38.503727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503715    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9692736c-0929-420f-8e8d-7e35565d5cb0" containerName="console"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503728    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9692736c-0929-420f-8e8d-7e35565d5cb0" containerName="console"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503739    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9a25f6b9-47c2-45c0-9333-239c7103e570" containerName="registry"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503744    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a25f6b9-47c2-45c0-9333-239c7103e570" containerName="registry"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503752    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8645d924-656d-4790-a0d1-511c42b60e2b" containerName="insights-gathering"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503757    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8645d924-656d-4790-a0d1-511c42b60e2b" containerName="insights-gathering"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503769    5042 cpu_manager.go:414] "containerMap: removing container" podUID="578aad49-7f4d-4150-ab54-bb2b9f74940e" containerName="registry"
Nov 02 17:34:38.503782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503774    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="578aad49-7f4d-4150-ab54-bb2b9f74940e" containerName="registry"
Nov 02 17:34:38.504055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503833    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="578aad49-7f4d-4150-ab54-bb2b9f74940e" containerName="registry"
Nov 02 17:34:38.504055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503860    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9692736c-0929-420f-8e8d-7e35565d5cb0" containerName="console"
Nov 02 17:34:38.504055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503867    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a25f6b9-47c2-45c0-9333-239c7103e570" containerName="registry"
Nov 02 17:34:38.504055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.503872    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8645d924-656d-4790-a0d1-511c42b60e2b" containerName="insights-gathering"
Nov 02 17:34:38.506627 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.506610    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.514347 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.514117    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fjcwd-h6mhp"]
Nov 02 17:34:38.575015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.574977    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbmsb\" (UniqueName: \"kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.575015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.575023    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.575245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.575095    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.575245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.575179    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.675869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.675801    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.676066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.675906    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.676066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.675947    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbmsb\" (UniqueName: \"kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.676066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.675983    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.676398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.676371    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.676576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.676554    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.676674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.676654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.688243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.688221    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbmsb\" (UniqueName: \"kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb\") pod \"periodic-gathering-fjcwd-h6mhp\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") " pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:38.820340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:38.820258    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:34:39.084030 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:39.083955    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-hk6xf" event={"ID":"52a1ad24-d5c4-49cf-aa71-b92f14918879","Type":"ContainerStarted","Data":"27f18b6b77e2ef3639f4ee41c1af96c72116fcce3df6b9106fa446e49b747d0e"}
Nov 02 17:34:39.092188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:39.092090    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fjcwd-h6mhp"]
Nov 02 17:34:39.094693 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:34:39.094670    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02abe3a_c622_428b_9650_bdfd6ebf85fd.slice/crio-ef5c0b633dafc9331e6ed4a670999e5ca45b589a09b4946886b124a7376b6612 WatchSource:0}: Error finding container ef5c0b633dafc9331e6ed4a670999e5ca45b589a09b4946886b124a7376b6612: Status 404 returned error can't find the container with id ef5c0b633dafc9331e6ed4a670999e5ca45b589a09b4946886b124a7376b6612
Nov 02 17:34:40.089614 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:40.089570    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp" event={"ID":"f02abe3a-c622-428b-9650-bdfd6ebf85fd","Type":"ContainerStarted","Data":"15b547c3f08ff206a334e72ea0e59e53c0b137c699a5f6a8c0154499f57ea233"}
Nov 02 17:34:40.090185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:40.089665    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp" event={"ID":"f02abe3a-c622-428b-9650-bdfd6ebf85fd","Type":"ContainerStarted","Data":"ef5c0b633dafc9331e6ed4a670999e5ca45b589a09b4946886b124a7376b6612"}
Nov 02 17:34:40.106305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:40.106254    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp" podStartSLOduration=2.106240985 podStartE2EDuration="2.106240985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:40.106081019 +0000 UTC m=+92.685490406" watchObservedRunningTime="2026-11-02 17:34:40.106240985 +0000 UTC m=+92.685650362"
Nov 02 17:34:43.103384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:43.103349    5042 generic.go:356] "Generic (PLEG): container finished" podID="246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c" containerID="37a51d35f174f94709456c92a2c901db773ac3e5a71812026af9cb88e2d15b46" exitCode=0
Nov 02 17:34:43.103899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:43.103425    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm" event={"ID":"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c","Type":"ContainerDied","Data":"37a51d35f174f94709456c92a2c901db773ac3e5a71812026af9cb88e2d15b46"}
Nov 02 17:34:43.103899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:43.103884    5042 scope.go:122] "RemoveContainer" containerID="37a51d35f174f94709456c92a2c901db773ac3e5a71812026af9cb88e2d15b46"
Nov 02 17:34:44.109444 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:34:44.109399    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-75jlm" event={"ID":"246bbed4-d3f9-4a1b-9458-a8a7cbeb3f7c","Type":"ContainerStarted","Data":"aacbfede69ab7e32088d2bb92117681fb67352964b0aeaeda6e772e85bc0765f"}
Nov 02 17:34:48.538417 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:34:48.538374    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:35:18.537606 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:35:18.537528    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:35:48.538407 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:35:48.538370    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:35:51.698718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.698688    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"]
Nov 02 17:35:51.702056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.702038    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.710771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.710749    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 17:35:51.715660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.715630    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"]
Nov 02 17:35:51.804183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.804149    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-5pxf7\""
Nov 02 17:35:51.805417 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805395    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805431    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805510    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805574 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805562    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805610 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805603    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbch\" (UniqueName: \"kubernetes.io/projected/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-kube-api-access-vlbch\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805634    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805665    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805700    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.805719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.805717    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.807675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.807649    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"]
Nov 02 17:35:51.811134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.811119    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:51.814910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.814891    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 17:35:51.819180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.819157    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"]
Nov 02 17:35:51.842809 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.842782    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-82dm4\""
Nov 02 17:35:51.906746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.906714    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.906746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.906750    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.906772    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.906894    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:51.907031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.906944    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907084    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4gl\" (UniqueName: \"kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:51.907156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907105    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907186    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907211    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907249    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbch\" (UniqueName: \"kubernetes.io/projected/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-kube-api-access-vlbch\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907273    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907326    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907331    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907355    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:51.907594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907509    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907533    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.907594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.907580    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.909717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.909691    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.909890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.909875    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.915658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.915637    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 17:35:51.915794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.915771    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:51.916100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:51.916081    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbch\" (UniqueName: \"kubernetes.io/projected/bc408cd5-d5f4-471f-b48d-1bc3b86eec74-kube-api-access-vlbch\") pod \"openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb\" (UID: \"bc408cd5-d5f4-471f-b48d-1bc3b86eec74\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:52.008397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.008314    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.008397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.008365    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.008397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.008392    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4gl\" (UniqueName: \"kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.008625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.008548    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"]
Nov 02 17:35:52.008705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.008687    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.008780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.008761    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.013391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.013376    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.016714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.016688    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4gl\" (UniqueName: \"kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.017774 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.017755    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:52.019353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.019317    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-vcjd6\""
Nov 02 17:35:52.020626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.020604    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"]
Nov 02 17:35:52.109168 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.109136    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.109168 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.109172    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.109395 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.109261    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvm9\" (UniqueName: \"kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.128650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.128624    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:35:52.210414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.210380    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.210414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.210418    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.210662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.210453    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvm9\" (UniqueName: \"kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.210768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.210748    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.210832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.210777    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.219395 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.219373    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvm9\" (UniqueName: \"kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.297149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.297123    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"]
Nov 02 17:35:52.299867 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:35:52.299821    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc408cd5_d5f4_471f_b48d_1bc3b86eec74.slice/crio-01d901a6a06da8239366b39ac365aa88665b05dbbfaecaeecd348fb25e4ac63d WatchSource:0}: Error finding container 01d901a6a06da8239366b39ac365aa88665b05dbbfaecaeecd348fb25e4ac63d: Status 404 returned error can't find the container with id 01d901a6a06da8239366b39ac365aa88665b05dbbfaecaeecd348fb25e4ac63d
Nov 02 17:35:52.357976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.357946    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:35:52.410244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.409558    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"]
Nov 02 17:35:52.415821 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:35:52.415783    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef588b4_6f74_4553_b1a7_03a321b6440f.slice/crio-987cd6c458ad6610c534eb759887ed08b24f9c2c9b27725074bab0429a444de5 WatchSource:0}: Error finding container 987cd6c458ad6610c534eb759887ed08b24f9c2c9b27725074bab0429a444de5: Status 404 returned error can't find the container with id 987cd6c458ad6610c534eb759887ed08b24f9c2c9b27725074bab0429a444de5
Nov 02 17:35:52.420127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.420092    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb" event={"ID":"bc408cd5-d5f4-471f-b48d-1bc3b86eec74","Type":"ContainerStarted","Data":"01d901a6a06da8239366b39ac365aa88665b05dbbfaecaeecd348fb25e4ac63d"}
Nov 02 17:35:52.631893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:52.631784    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"]
Nov 02 17:35:52.634743 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:35:52.634712    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42c23b3_881c_49b6_8f9d_58d7e8febe71.slice/crio-990b79611d1c609a5674f279a6413c06f46f8ab227e1b1b09668b4f3d40b94fd WatchSource:0}: Error finding container 990b79611d1c609a5674f279a6413c06f46f8ab227e1b1b09668b4f3d40b94fd: Status 404 returned error can't find the container with id 990b79611d1c609a5674f279a6413c06f46f8ab227e1b1b09668b4f3d40b94fd
Nov 02 17:35:53.428180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:53.428136    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2" event={"ID":"5ef588b4-6f74-4553-b1a7-03a321b6440f","Type":"ContainerStarted","Data":"987cd6c458ad6610c534eb759887ed08b24f9c2c9b27725074bab0429a444de5"}
Nov 02 17:35:53.430407 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:53.430363    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn" event={"ID":"c42c23b3-881c-49b6-8f9d-58d7e8febe71","Type":"ContainerStarted","Data":"990b79611d1c609a5674f279a6413c06f46f8ab227e1b1b09668b4f3d40b94fd"}
Nov 02 17:35:56.479959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:56.479919    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:56.480377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:56.479998    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:56.480377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:56.480028    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:35:57.457374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:57.457333    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb" event={"ID":"bc408cd5-d5f4-471f-b48d-1bc3b86eec74","Type":"ContainerStarted","Data":"9ddeda454d48234960318f37f0dd6825ab9641b44a61e421b55ad87961732340"}
Nov 02 17:35:57.491797 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:57.491704    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb" podStartSLOduration=2.31446048 podStartE2EDuration="6.491682401s" totalImagesPullingTime="4.177221921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:57.489570719 +0000 UTC m=+170.068980106" watchObservedRunningTime="2026-11-02 17:35:57.491682401 +0000 UTC m=+170.071091788"
Nov 02 17:35:58.018351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:58.018263    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:58.024311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:58.024286    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:58.463513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:58.463483    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:58.464724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:58.464491    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb"
Nov 02 17:35:59.468404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:59.468376    5042 generic.go:356] "Generic (PLEG): container finished" podID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerID="181c51599ea7bd5f15f4a9d0dc623803adbbd1bb5627df5de2730367088b728d" exitCode=0
Nov 02 17:35:59.468799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:35:59.468454    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2" event={"ID":"5ef588b4-6f74-4553-b1a7-03a321b6440f","Type":"ContainerDied","Data":"181c51599ea7bd5f15f4a9d0dc623803adbbd1bb5627df5de2730367088b728d"}
Nov 02 17:36:00.474588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:00.474553    5042 generic.go:356] "Generic (PLEG): container finished" podID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerID="559e192ed6e46028316283a22813298d570f05a0d844faafedb03de7cedfcd04" exitCode=0
Nov 02 17:36:00.475097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:00.474644    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn" event={"ID":"c42c23b3-881c-49b6-8f9d-58d7e8febe71","Type":"ContainerDied","Data":"559e192ed6e46028316283a22813298d570f05a0d844faafedb03de7cedfcd04"}
Nov 02 17:36:01.480969 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:01.480933    5042 generic.go:356] "Generic (PLEG): container finished" podID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerID="6923fe43c53ecbe40e40c5c97c9b6f2a0d42f6f014a43b06cb276c9f7417f9c9" exitCode=0
Nov 02 17:36:01.481386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:01.480978    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn" event={"ID":"c42c23b3-881c-49b6-8f9d-58d7e8febe71","Type":"ContainerDied","Data":"6923fe43c53ecbe40e40c5c97c9b6f2a0d42f6f014a43b06cb276c9f7417f9c9"}
Nov 02 17:36:02.488398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:02.488360    5042 generic.go:356] "Generic (PLEG): container finished" podID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerID="b86f801e5c6a3b3baff59d02c49baa43e6e84d68ea9ad7ee2a61a14eac86e9c5" exitCode=0
Nov 02 17:36:02.488878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:02.488433    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2" event={"ID":"5ef588b4-6f74-4553-b1a7-03a321b6440f","Type":"ContainerDied","Data":"b86f801e5c6a3b3baff59d02c49baa43e6e84d68ea9ad7ee2a61a14eac86e9c5"}
Nov 02 17:36:03.495909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:03.495868    5042 generic.go:356] "Generic (PLEG): container finished" podID="f02abe3a-c622-428b-9650-bdfd6ebf85fd" containerID="15b547c3f08ff206a334e72ea0e59e53c0b137c699a5f6a8c0154499f57ea233" exitCode=0
Nov 02 17:36:03.495909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:03.495886    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp" event={"ID":"f02abe3a-c622-428b-9650-bdfd6ebf85fd","Type":"ContainerDied","Data":"15b547c3f08ff206a334e72ea0e59e53c0b137c699a5f6a8c0154499f57ea233"}
Nov 02 17:36:04.672611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.672583    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:36:04.739369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.738832    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path\") pod \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") "
Nov 02 17:36:04.739369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.738909    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle\") pod \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") "
Nov 02 17:36:04.739369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.738956    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle\") pod \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") "
Nov 02 17:36:04.739369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.739038    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb\" (UniqueName: \"kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb\") pod \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\" (UID: \"f02abe3a-c622-428b-9650-bdfd6ebf85fd\") "
Nov 02 17:36:04.739369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.739156    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path" pod "f02abe3a-c622-428b-9650-bdfd6ebf85fd" (UID: "f02abe3a-c622-428b-9650-bdfd6ebf85fd"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:36:04.739738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.739486    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle" pod "f02abe3a-c622-428b-9650-bdfd6ebf85fd" (UID: "f02abe3a-c622-428b-9650-bdfd6ebf85fd"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:36:04.740271 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.740243    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle" pod "f02abe3a-c622-428b-9650-bdfd6ebf85fd" (UID: "f02abe3a-c622-428b-9650-bdfd6ebf85fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:36:04.742316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.742281    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb" pod "f02abe3a-c622-428b-9650-bdfd6ebf85fd" (UID: "f02abe3a-c622-428b-9650-bdfd6ebf85fd"). InnerVolumeSpecName "kube-api-access-fbmsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:36:04.840107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.840021    5042 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-trusted-ca-bundle\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:04.840107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.840059    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbmsb\" (UniqueName: \"kubernetes.io/projected/f02abe3a-c622-428b-9650-bdfd6ebf85fd-kube-api-access-fbmsb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:04.840107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.840077    5042 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f02abe3a-c622-428b-9650-bdfd6ebf85fd-archives-path\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:04.840107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:04.840090    5042 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f02abe3a-c622-428b-9650-bdfd6ebf85fd-service-ca-bundle\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:05.508454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:05.508423    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp"
Nov 02 17:36:05.508630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:05.508415    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fjcwd-h6mhp" event={"ID":"f02abe3a-c622-428b-9650-bdfd6ebf85fd","Type":"ContainerDied","Data":"ef5c0b633dafc9331e6ed4a670999e5ca45b589a09b4946886b124a7376b6612"}
Nov 02 17:36:05.508630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:05.508600    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef5c0b633dafc9331e6ed4a670999e5ca45b589a09b4946886b124a7376b6612"
Nov 02 17:36:08.525780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:08.525739    5042 generic.go:356] "Generic (PLEG): container finished" podID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerID="2db50b538c8d9f998efccd1a821769ff8cbf3c134c38cec3b09701b77691ddcb" exitCode=0
Nov 02 17:36:08.526249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:08.525827    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn" event={"ID":"c42c23b3-881c-49b6-8f9d-58d7e8febe71","Type":"ContainerDied","Data":"2db50b538c8d9f998efccd1a821769ff8cbf3c134c38cec3b09701b77691ddcb"}
Nov 02 17:36:09.532496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.532461    5042 generic.go:356] "Generic (PLEG): container finished" podID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerID="197c00bd2b9f8ee5ce23bc988bda9653842350181689b0065a7936fb508ca2d3" exitCode=0
Nov 02 17:36:09.532919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.532551    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2" event={"ID":"5ef588b4-6f74-4553-b1a7-03a321b6440f","Type":"ContainerDied","Data":"197c00bd2b9f8ee5ce23bc988bda9653842350181689b0065a7936fb508ca2d3"}
Nov 02 17:36:09.662101 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.662072    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:36:09.687180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.687146    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle\") pod \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") "
Nov 02 17:36:09.687349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.687244    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9\" (UniqueName: \"kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9\") pod \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") "
Nov 02 17:36:09.687419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.687359    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util\") pod \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\" (UID: \"c42c23b3-881c-49b6-8f9d-58d7e8febe71\") "
Nov 02 17:36:09.687716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.687692    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle" pod "c42c23b3-881c-49b6-8f9d-58d7e8febe71" (UID: "c42c23b3-881c-49b6-8f9d-58d7e8febe71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:36:09.689639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.689606    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9" pod "c42c23b3-881c-49b6-8f9d-58d7e8febe71" (UID: "c42c23b3-881c-49b6-8f9d-58d7e8febe71"). InnerVolumeSpecName "kube-api-access-jjvm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:36:09.694583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.694551    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util" pod "c42c23b3-881c-49b6-8f9d-58d7e8febe71" (UID: "c42c23b3-881c-49b6-8f9d-58d7e8febe71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:36:09.788322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.788233    5042 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-util\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:09.788322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.788266    5042 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c42c23b3-881c-49b6-8f9d-58d7e8febe71-bundle\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:09.788322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:09.788278    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjvm9\" (UniqueName: \"kubernetes.io/projected/c42c23b3-881c-49b6-8f9d-58d7e8febe71-kube-api-access-jjvm9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:10.538788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.538763    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn"
Nov 02 17:36:10.539272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.538793    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cg2kn" event={"ID":"c42c23b3-881c-49b6-8f9d-58d7e8febe71","Type":"ContainerDied","Data":"990b79611d1c609a5674f279a6413c06f46f8ab227e1b1b09668b4f3d40b94fd"}
Nov 02 17:36:10.539272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.538850    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990b79611d1c609a5674f279a6413c06f46f8ab227e1b1b09668b4f3d40b94fd"
Nov 02 17:36:10.669004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.668974    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:36:10.695036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.695004    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util\") pod \"5ef588b4-6f74-4553-b1a7-03a321b6440f\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") "
Nov 02 17:36:10.695192 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.695071    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl\" (UniqueName: \"kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl\") pod \"5ef588b4-6f74-4553-b1a7-03a321b6440f\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") "
Nov 02 17:36:10.695192 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.695094    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle\") pod \"5ef588b4-6f74-4553-b1a7-03a321b6440f\" (UID: \"5ef588b4-6f74-4553-b1a7-03a321b6440f\") "
Nov 02 17:36:10.695737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.695710    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle" pod "5ef588b4-6f74-4553-b1a7-03a321b6440f" (UID: "5ef588b4-6f74-4553-b1a7-03a321b6440f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:36:10.697469 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.697441    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl" pod "5ef588b4-6f74-4553-b1a7-03a321b6440f" (UID: "5ef588b4-6f74-4553-b1a7-03a321b6440f"). InnerVolumeSpecName "kube-api-access-zt4gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:36:10.700122 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.700097    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util" pod "5ef588b4-6f74-4553-b1a7-03a321b6440f" (UID: "5ef588b4-6f74-4553-b1a7-03a321b6440f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:36:10.796699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.796611    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4gl\" (UniqueName: \"kubernetes.io/projected/5ef588b4-6f74-4553-b1a7-03a321b6440f-kube-api-access-zt4gl\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:10.796699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.796644    5042 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-bundle\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:10.796699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:10.796656    5042 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ef588b4-6f74-4553-b1a7-03a321b6440f-util\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:36:11.544442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:11.544416    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2"
Nov 02 17:36:11.544442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:11.544426    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c7nlx2" event={"ID":"5ef588b4-6f74-4553-b1a7-03a321b6440f","Type":"ContainerDied","Data":"987cd6c458ad6610c534eb759887ed08b24f9c2c9b27725074bab0429a444de5"}
Nov 02 17:36:11.545013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:11.544463    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="987cd6c458ad6610c534eb759887ed08b24f9c2c9b27725074bab0429a444de5"
Nov 02 17:36:18.537083 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:36:18.537043    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:36:19.159137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159105    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"]
Nov 02 17:36:19.159739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159711    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="extract"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159740    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="extract"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159773    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="pull"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159780    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="pull"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159798    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="util"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159804    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="util"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159812    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f02abe3a-c622-428b-9650-bdfd6ebf85fd" containerName="insights-gathering"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159817    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02abe3a-c622-428b-9650-bdfd6ebf85fd" containerName="insights-gathering"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159827    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="util"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159833    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="util"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159868    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="extract"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159877    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="extract"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159894    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="pull"
Nov 02 17:36:19.159895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159902    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="pull"
Nov 02 17:36:19.160523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159981    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c42c23b3-881c-49b6-8f9d-58d7e8febe71" containerName="extract"
Nov 02 17:36:19.160523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.159991    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef588b4-6f74-4553-b1a7-03a321b6440f" containerName="extract"
Nov 02 17:36:19.160523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.160001    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02abe3a-c622-428b-9650-bdfd6ebf85fd" containerName="insights-gathering"
Nov 02 17:36:19.181440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.181409    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.186683 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.186663    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-z8prg\""
Nov 02 17:36:19.186814 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.186778    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 17:36:19.202121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.202080    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"]
Nov 02 17:36:19.269550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.269513    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.269722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.269577    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.269765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.269711    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.269818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.269801    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.269877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.269848    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl5mw\" (UniqueName: \"kubernetes.io/projected/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-kube-api-access-gl5mw\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.269877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.269869    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.281508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.281477    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 17:36:19.281668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.281622    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 17:36:19.281721 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.281693    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 17:36:19.282619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.282606    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 17:36:19.371073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371036    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.371257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371090    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.371257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371139    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.371257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371165    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl5mw\" (UniqueName: \"kubernetes.io/projected/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-kube-api-access-gl5mw\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.371398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371273    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.371398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371330    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.371800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.371765    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-manifests\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.373702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.373677    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-tmp\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.373859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.373826    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.373946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.373929    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-webhook-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.374078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.374059    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-metrics-cert\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.385246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.385212    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl5mw\" (UniqueName: \"kubernetes.io/projected/2e8003ad-c2d2-483b-8ccf-d48aadd55aed-kube-api-access-gl5mw\") pod \"opendatahub-operator-controller-manager-789486f7c8-sx6g4\" (UID: \"2e8003ad-c2d2-483b-8ccf-d48aadd55aed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.518253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.518209    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:19.844514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:19.844486    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"]
Nov 02 17:36:19.846342 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:19.846317    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8003ad_c2d2_483b_8ccf_d48aadd55aed.slice/crio-7197bce36220358fb43b7daae0c118ad25df5a427cbe4f3fa1ac8023912b4e99 WatchSource:0}: Error finding container 7197bce36220358fb43b7daae0c118ad25df5a427cbe4f3fa1ac8023912b4e99: Status 404 returned error can't find the container with id 7197bce36220358fb43b7daae0c118ad25df5a427cbe4f3fa1ac8023912b4e99
Nov 02 17:36:20.584825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:20.584783    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4" event={"ID":"2e8003ad-c2d2-483b-8ccf-d48aadd55aed","Type":"ContainerStarted","Data":"7197bce36220358fb43b7daae0c118ad25df5a427cbe4f3fa1ac8023912b4e99"}
Nov 02 17:36:23.600912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:23.600875    5042 generic.go:356] "Generic (PLEG): container finished" podID="2e8003ad-c2d2-483b-8ccf-d48aadd55aed" containerID="1d59873298a6664cc00c675b788100a030b832a4d5d185ebbf25665eb4b33850" exitCode=0
Nov 02 17:36:23.601327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:23.600954    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4" event={"ID":"2e8003ad-c2d2-483b-8ccf-d48aadd55aed","Type":"ContainerDied","Data":"1d59873298a6664cc00c675b788100a030b832a4d5d185ebbf25665eb4b33850"}
Nov 02 17:36:24.607625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:24.607575    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4" event={"ID":"2e8003ad-c2d2-483b-8ccf-d48aadd55aed","Type":"ContainerStarted","Data":"980818752ba4e6a1fb78ddf6877b5d06f8ecf0a3fc31fe7a5932a5a304b598b2"}
Nov 02 17:36:24.608032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:24.607780    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:24.769126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:24.769062    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4" podStartSLOduration=1.74415851 podStartE2EDuration="5.769043639s" totalImagesPullingTime="3.498604445s" totalInitContainerRuntime="526.280684ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:24.765207265 +0000 UTC m=+197.344616650" watchObservedRunningTime="2026-11-02 17:36:24.769043639 +0000 UTC m=+197.348453029"
Nov 02 17:36:34.432455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.432418    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"]
Nov 02 17:36:34.438907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.438878    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:34.447077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.447049    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ggtx6\""
Nov 02 17:36:34.447216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.447114    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 17:36:34.447216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.447171    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 17:36:34.448808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.448788    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"]
Nov 02 17:36:34.507497 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.507466    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7tqv\" (UniqueName: \"kubernetes.io/projected/06bdf526-bccd-4303-b9f5-ff3131a4ae96-kube-api-access-w7tqv\") pod \"authorino-operator-7b7bb4bc77-lx6js\" (UID: \"06bdf526-bccd-4303-b9f5-ff3131a4ae96\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:34.608632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.608596    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7tqv\" (UniqueName: \"kubernetes.io/projected/06bdf526-bccd-4303-b9f5-ff3131a4ae96-kube-api-access-w7tqv\") pod \"authorino-operator-7b7bb4bc77-lx6js\" (UID: \"06bdf526-bccd-4303-b9f5-ff3131a4ae96\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:34.621205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.621179    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7tqv\" (UniqueName: \"kubernetes.io/projected/06bdf526-bccd-4303-b9f5-ff3131a4ae96-kube-api-access-w7tqv\") pod \"authorino-operator-7b7bb4bc77-lx6js\" (UID: \"06bdf526-bccd-4303-b9f5-ff3131a4ae96\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:34.752788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:34.752756    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:35.026286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:35.026262    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"]
Nov 02 17:36:35.028932 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:35.028904    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bdf526_bccd_4303_b9f5_ff3131a4ae96.slice/crio-4cd340d1b4b6b49c0b86c6ce51def159a790c4dbd657dca7dca04ef1c400120a WatchSource:0}: Error finding container 4cd340d1b4b6b49c0b86c6ce51def159a790c4dbd657dca7dca04ef1c400120a: Status 404 returned error can't find the container with id 4cd340d1b4b6b49c0b86c6ce51def159a790c4dbd657dca7dca04ef1c400120a
Nov 02 17:36:35.615002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:35.614975    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4"
Nov 02 17:36:35.655965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:35.655934    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js" event={"ID":"06bdf526-bccd-4303-b9f5-ff3131a4ae96","Type":"ContainerStarted","Data":"4cd340d1b4b6b49c0b86c6ce51def159a790c4dbd657dca7dca04ef1c400120a"}
Nov 02 17:36:38.050398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.050356    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"]
Nov 02 17:36:38.062443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.062415    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:38.064230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.064196    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"]
Nov 02 17:36:38.067396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.067372    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 02 17:36:38.067533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.067410    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x7ddr\""
Nov 02 17:36:38.141642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.141612    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz9zc\" (UniqueName: \"kubernetes.io/projected/d56437ee-40bb-439d-812f-1a55863eac69-kube-api-access-dz9zc\") pod \"dns-operator-controller-manager-8557cbdb84-k9mfj\" (UID: \"d56437ee-40bb-439d-812f-1a55863eac69\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:38.243146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.243106    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz9zc\" (UniqueName: \"kubernetes.io/projected/d56437ee-40bb-439d-812f-1a55863eac69-kube-api-access-dz9zc\") pod \"dns-operator-controller-manager-8557cbdb84-k9mfj\" (UID: \"d56437ee-40bb-439d-812f-1a55863eac69\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:38.252228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.252195    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz9zc\" (UniqueName: \"kubernetes.io/projected/d56437ee-40bb-439d-812f-1a55863eac69-kube-api-access-dz9zc\") pod \"dns-operator-controller-manager-8557cbdb84-k9mfj\" (UID: \"d56437ee-40bb-439d-812f-1a55863eac69\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:38.379719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.379632    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:38.671577 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.671493    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js" event={"ID":"06bdf526-bccd-4303-b9f5-ff3131a4ae96","Type":"ContainerStarted","Data":"378e1089aedd84b7ca2ae708712c118f4e0ea4f59bd3a4a0428a0c59b93976f1"}
Nov 02 17:36:38.671729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.671624    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:38.678676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.678644    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"]
Nov 02 17:36:38.680574 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:38.680550    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56437ee_40bb_439d_812f_1a55863eac69.slice/crio-1d4a31606c7fa1705d425c0cf18a2cbe7cda8119cab4d15f736340817d1e3c45 WatchSource:0}: Error finding container 1d4a31606c7fa1705d425c0cf18a2cbe7cda8119cab4d15f736340817d1e3c45: Status 404 returned error can't find the container with id 1d4a31606c7fa1705d425c0cf18a2cbe7cda8119cab4d15f736340817d1e3c45
Nov 02 17:36:38.691465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:38.691419    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js" podStartSLOduration=1.21163651 podStartE2EDuration="4.691406285s" totalImagesPullingTime="3.479769775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:38.690402867 +0000 UTC m=+211.269812253" watchObservedRunningTime="2026-11-02 17:36:38.691406285 +0000 UTC m=+211.270815671"
Nov 02 17:36:39.676751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:39.676716    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj" event={"ID":"d56437ee-40bb-439d-812f-1a55863eac69","Type":"ContainerStarted","Data":"1d4a31606c7fa1705d425c0cf18a2cbe7cda8119cab4d15f736340817d1e3c45"}
Nov 02 17:36:41.420189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.420154    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"]
Nov 02 17:36:41.424013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.423993    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:41.433086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.433059    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-f8vn2\""
Nov 02 17:36:41.450115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.450088    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"]
Nov 02 17:36:41.573928 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.573890    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddvph\" (UniqueName: \"kubernetes.io/projected/5488c1fe-6a3d-464a-bd15-2377b7740b9b-kube-api-access-ddvph\") pod \"limitador-operator-controller-manager-5778bb4449-ljs7q\" (UID: \"5488c1fe-6a3d-464a-bd15-2377b7740b9b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:41.674690 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.674601    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddvph\" (UniqueName: \"kubernetes.io/projected/5488c1fe-6a3d-464a-bd15-2377b7740b9b-kube-api-access-ddvph\") pod \"limitador-operator-controller-manager-5778bb4449-ljs7q\" (UID: \"5488c1fe-6a3d-464a-bd15-2377b7740b9b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:41.687343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.687308    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddvph\" (UniqueName: \"kubernetes.io/projected/5488c1fe-6a3d-464a-bd15-2377b7740b9b-kube-api-access-ddvph\") pod \"limitador-operator-controller-manager-5778bb4449-ljs7q\" (UID: \"5488c1fe-6a3d-464a-bd15-2377b7740b9b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:41.737674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:41.737632    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:42.022330 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:42.022297    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"]
Nov 02 17:36:42.024582 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:42.024547    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5488c1fe_6a3d_464a_bd15_2377b7740b9b.slice/crio-291ace255bd976884ac62d2f3f4d21d14ed0c8ef64a87c1d7fa617af39c653a8 WatchSource:0}: Error finding container 291ace255bd976884ac62d2f3f4d21d14ed0c8ef64a87c1d7fa617af39c653a8: Status 404 returned error can't find the container with id 291ace255bd976884ac62d2f3f4d21d14ed0c8ef64a87c1d7fa617af39c653a8
Nov 02 17:36:42.691510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:42.691471    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q" event={"ID":"5488c1fe-6a3d-464a-bd15-2377b7740b9b","Type":"ContainerStarted","Data":"291ace255bd976884ac62d2f3f4d21d14ed0c8ef64a87c1d7fa617af39c653a8"}
Nov 02 17:36:44.704271 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:44.704240    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q" event={"ID":"5488c1fe-6a3d-464a-bd15-2377b7740b9b","Type":"ContainerStarted","Data":"219f281d4060005bb62ece2053630db6304d5e049aac287ddcd757c21a754d50"}
Nov 02 17:36:44.704706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:44.704302    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:44.705938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:44.705902    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj" event={"ID":"d56437ee-40bb-439d-812f-1a55863eac69","Type":"ContainerStarted","Data":"31cc061e36e9a01eff958da8d03c681a9b23ad4ecc4ae9cbcb9f4dc86e48969d"}
Nov 02 17:36:44.706094 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:44.705971    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:44.723952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:44.723908    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q" podStartSLOduration=2.0879313 podStartE2EDuration="3.723893548s" totalImagesPullingTime="1.635962248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:44.722256869 +0000 UTC m=+217.301666252" watchObservedRunningTime="2026-11-02 17:36:44.723893548 +0000 UTC m=+217.303302934"
Nov 02 17:36:44.741629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:44.741588    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj" podStartSLOduration=1.097960603 podStartE2EDuration="6.741575649s" totalImagesPullingTime="5.643615046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:44.740355987 +0000 UTC m=+217.319765373" watchObservedRunningTime="2026-11-02 17:36:44.741575649 +0000 UTC m=+217.320985034"
Nov 02 17:36:46.074102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.074063    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"]
Nov 02 17:36:46.077617 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.077600    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.083694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.083665    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2v2nf\""
Nov 02 17:36:46.093265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.093243    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"]
Nov 02 17:36:46.216217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.216182    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npscp\" (UniqueName: \"kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp\") pod \"kuadrant-operator-controller-manager-678d87766f-f56vl\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.216396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.216236    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f56vl\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.317700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.317654    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npscp\" (UniqueName: \"kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp\") pod \"kuadrant-operator-controller-manager-678d87766f-f56vl\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.317914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.317713    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f56vl\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.318077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.318056    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f56vl\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.334684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.334619    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npscp\" (UniqueName: \"kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp\") pod \"kuadrant-operator-controller-manager-678d87766f-f56vl\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.390172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.390141    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:46.686384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.686072    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"]
Nov 02 17:36:46.688770 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:46.688729    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588a37ce_4b38_4224_9f67_adb7273f76b5.slice/crio-f5f4e67eb6430282de17ae512983e8735f0103d0cabffcb48267ca6b5c50953e WatchSource:0}: Error finding container f5f4e67eb6430282de17ae512983e8735f0103d0cabffcb48267ca6b5c50953e: Status 404 returned error can't find the container with id f5f4e67eb6430282de17ae512983e8735f0103d0cabffcb48267ca6b5c50953e
Nov 02 17:36:46.717208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:46.717173    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" event={"ID":"588a37ce-4b38-4224-9f67-adb7273f76b5","Type":"ContainerStarted","Data":"f5f4e67eb6430282de17ae512983e8735f0103d0cabffcb48267ca6b5c50953e"}
Nov 02 17:36:48.544655 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:36:48.544562    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:36:49.680678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:49.679975    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js"
Nov 02 17:36:51.743156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:51.743115    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" event={"ID":"588a37ce-4b38-4224-9f67-adb7273f76b5","Type":"ContainerStarted","Data":"f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4"}
Nov 02 17:36:51.743613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:51.743201    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:36:51.770684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:51.770628    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" podStartSLOduration=1.741747785 podStartE2EDuration="5.770614223s" totalImagesPullingTime="4.028866438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:51.767764236 +0000 UTC m=+224.347173621" watchObservedRunningTime="2026-11-02 17:36:51.770614223 +0000 UTC m=+224.350023608"
Nov 02 17:36:53.378716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.378676    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"]
Nov 02 17:36:53.383963 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.383933    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.397035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.397012    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-pmq24\""
Nov 02 17:36:53.455112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.455062    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"]
Nov 02 17:36:53.582524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.582478    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.582777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.582755    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.582886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.582790    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.583014 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.582987    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.583051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.583031    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dd5948-ec52-4943-b15d-312a915f9b3f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.583140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.583115    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.583206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.583167    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.583314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.583296    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.583354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.583339    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5d7j\" (UniqueName: \"kubernetes.io/projected/c6dd5948-ec52-4943-b15d-312a915f9b3f-kube-api-access-z5d7j\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.683930    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.683992    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684024    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684092    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684119    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dd5948-ec52-4943-b15d-312a915f9b3f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684151    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684181    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684237    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684269    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5d7j\" (UniqueName: \"kubernetes.io/projected/c6dd5948-ec52-4943-b15d-312a915f9b3f-kube-api-access-z5d7j\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684496    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684533    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684793    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.684966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684899    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.685025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.684972    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dd5948-ec52-4943-b15d-312a915f9b3f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.686795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.686772    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.687128 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.687108    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.709322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.709276    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dd5948-ec52-4943-b15d-312a915f9b3f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:53.709501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:53.709478    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5d7j\" (UniqueName: \"kubernetes.io/projected/c6dd5948-ec52-4943-b15d-312a915f9b3f-kube-api-access-z5d7j\") pod \"data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8\" (UID: \"c6dd5948-ec52-4943-b15d-312a915f9b3f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:54.000490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.000453    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:54.351920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.351892    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"]
Nov 02 17:36:54.352589 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:54.352545    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6dd5948_ec52_4943_b15d_312a915f9b3f.slice/crio-5b31b0926e22f69d8f7ec32861dc0c94a5669b846c57574888a88b17ddaba6e6 WatchSource:0}: Error finding container 5b31b0926e22f69d8f7ec32861dc0c94a5669b846c57574888a88b17ddaba6e6: Status 404 returned error can't find the container with id 5b31b0926e22f69d8f7ec32861dc0c94a5669b846c57574888a88b17ddaba6e6
Nov 02 17:36:54.354795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.354762    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:36:54.354918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.354877    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:36:54.354978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.354936    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:36:54.760205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.760172    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8" event={"ID":"c6dd5948-ec52-4943-b15d-312a915f9b3f","Type":"ContainerStarted","Data":"29ea16b029506627099d2339624c0c6aef5bcb9bbd2fd36a144b33391b1a8784"}
Nov 02 17:36:54.760577 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.760253    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8" event={"ID":"c6dd5948-ec52-4943-b15d-312a915f9b3f","Type":"ContainerStarted","Data":"5b31b0926e22f69d8f7ec32861dc0c94a5669b846c57574888a88b17ddaba6e6"}
Nov 02 17:36:54.809565 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:54.809504    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8" podStartSLOduration=1.809485371 podStartE2EDuration="1.809485371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:54.806742361 +0000 UTC m=+227.386151746" watchObservedRunningTime="2026-11-02 17:36:54.809485371 +0000 UTC m=+227.388894763"
Nov 02 17:36:55.000756 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.000722    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:55.005601 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.005575    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:55.150798 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.150705    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4sbhs"]
Nov 02 17:36:55.153634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.153610    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:36:55.159027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.158995    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qjnd2\""
Nov 02 17:36:55.165237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.165213    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4sbhs"]
Nov 02 17:36:55.198924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.198883    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pc4\" (UniqueName: \"kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4\") pod \"authorino-57f7f8659f-4sbhs\" (UID: \"ec790129-59f9-4030-b260-23b2f7fcfc45\") " pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:36:55.280523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.280487    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"]
Nov 02 17:36:55.283023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.283003    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.291081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.291051    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 17:36:55.291200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.291055    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 17:36:55.291200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.291109    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-fbprk\""
Nov 02 17:36:55.294475 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.294452    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 17:36:55.299911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.299887    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pc4\" (UniqueName: \"kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4\") pod \"authorino-57f7f8659f-4sbhs\" (UID: \"ec790129-59f9-4030-b260-23b2f7fcfc45\") " pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:36:55.300029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.299919    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.300955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.300060    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.300955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.300201    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.300955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.300245    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkj72\" (UniqueName: \"kubernetes.io/projected/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-kube-api-access-hkj72\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.302439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.302418    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"]
Nov 02 17:36:55.312517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.312491    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pc4\" (UniqueName: \"kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4\") pod \"authorino-57f7f8659f-4sbhs\" (UID: \"ec790129-59f9-4030-b260-23b2f7fcfc45\") " pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:36:55.400671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.400630    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.400923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.400680    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkj72\" (UniqueName: \"kubernetes.io/projected/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-kube-api-access-hkj72\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.400923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.400724    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.400923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.400749    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.401152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.401134    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.401236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.401210    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.401601 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.401584    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.413010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.412967    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkj72\" (UniqueName: \"kubernetes.io/projected/6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0-kube-api-access-hkj72\") pod \"kserve-module-controller-manager-7566cf5fc8-tfbt9\" (UID: \"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.476761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.476722    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:36:55.504022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.503983    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"]
Nov 02 17:36:55.507891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.507868    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.522031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.521997    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"]
Nov 02 17:36:55.572726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.572696    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 17:36:55.572726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.572694    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 17:36:55.572936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.572749    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-gv62m\""
Nov 02 17:36:55.594993 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.594963    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:55.602116 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.602092    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-manager-config\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.602225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.602150    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkbq\" (UniqueName: \"kubernetes.io/projected/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-kube-api-access-rkkbq\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.603015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.602986    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-metrics-certs\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.603126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.603025    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-cert\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.604810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.604789    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:36:55.604810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.604790    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 17:36:55.604951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.604832    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 17:36:55.704605 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.704573    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-manager-config\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.704780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.704635    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkbq\" (UniqueName: \"kubernetes.io/projected/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-kube-api-access-rkkbq\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.704780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.704665    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-metrics-certs\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.704780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.704688    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-cert\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.705650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.705575    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-manager-config\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.707613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.707593    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-metrics-certs\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.707733 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.707647    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-cert\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.713248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.713225    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj"
Nov 02 17:36:55.713368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.713295    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q"
Nov 02 17:36:55.714465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.714431    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkbq\" (UniqueName: \"kubernetes.io/projected/f7ff2363-9be0-4573-a0e5-f4b843c0bb9b-kube-api-access-rkkbq\") pod \"jobset-controller-manager-6d9b8445f8-mbx87\" (UID: \"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b\") " pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.768041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.767554    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:55.768972 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.768950    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8"
Nov 02 17:36:55.771070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.771045    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4sbhs"]
Nov 02 17:36:55.772809 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:55.772778    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec790129_59f9_4030_b260_23b2f7fcfc45.slice/crio-fae7f2dcdc2b5b19f9cfa4c5685e4263531b52e35e563d06a4c833e0904696af WatchSource:0}: Error finding container fae7f2dcdc2b5b19f9cfa4c5685e4263531b52e35e563d06a4c833e0904696af: Status 404 returned error can't find the container with id fae7f2dcdc2b5b19f9cfa4c5685e4263531b52e35e563d06a4c833e0904696af
Nov 02 17:36:55.824694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.824659    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:55.882660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:55.882630    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"]
Nov 02 17:36:55.886254 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:55.886179    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e4902_3ee4_40f1_a1d5_a7720cf9d7a0.slice/crio-0a64398e41b9df8b02852deffc2a98c796f500e65e4c229d89fb72b0cd1b693f WatchSource:0}: Error finding container 0a64398e41b9df8b02852deffc2a98c796f500e65e4c229d89fb72b0cd1b693f: Status 404 returned error can't find the container with id 0a64398e41b9df8b02852deffc2a98c796f500e65e4c229d89fb72b0cd1b693f
Nov 02 17:36:56.129414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:56.129361    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"]
Nov 02 17:36:56.132126 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:36:56.132100    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ff2363_9be0_4573_a0e5_f4b843c0bb9b.slice/crio-597a2b8bbad31dff60e81104c43e869dd7f604a0ef08a1792659eb33d79fb5ea WatchSource:0}: Error finding container 597a2b8bbad31dff60e81104c43e869dd7f604a0ef08a1792659eb33d79fb5ea: Status 404 returned error can't find the container with id 597a2b8bbad31dff60e81104c43e869dd7f604a0ef08a1792659eb33d79fb5ea
Nov 02 17:36:56.775506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:56.775399    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87" event={"ID":"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b","Type":"ContainerStarted","Data":"597a2b8bbad31dff60e81104c43e869dd7f604a0ef08a1792659eb33d79fb5ea"}
Nov 02 17:36:56.777149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:56.777108    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9" event={"ID":"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0","Type":"ContainerStarted","Data":"0a64398e41b9df8b02852deffc2a98c796f500e65e4c229d89fb72b0cd1b693f"}
Nov 02 17:36:56.778641 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:56.778614    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4sbhs" event={"ID":"ec790129-59f9-4030-b260-23b2f7fcfc45","Type":"ContainerStarted","Data":"fae7f2dcdc2b5b19f9cfa4c5685e4263531b52e35e563d06a4c833e0904696af"}
Nov 02 17:36:58.809086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:58.809003    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4sbhs" event={"ID":"ec790129-59f9-4030-b260-23b2f7fcfc45","Type":"ContainerStarted","Data":"043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f"}
Nov 02 17:36:58.827130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:58.825958    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4sbhs" podStartSLOduration=1.13926216 podStartE2EDuration="3.825939775s" totalImagesPullingTime="2.686677615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:58.825597128 +0000 UTC m=+231.405006515" watchObservedRunningTime="2026-11-02 17:36:58.825939775 +0000 UTC m=+231.405349163"
Nov 02 17:36:59.519236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.519196    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4sbhs"]
Nov 02 17:36:59.815944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.815892    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87" event={"ID":"f7ff2363-9be0-4573-a0e5-f4b843c0bb9b","Type":"ContainerStarted","Data":"f44defe2ffb51848c7b7600d8bc97bd140b3707774a2968b6b899ef4eb60099e"}
Nov 02 17:36:59.816418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.816038    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:36:59.817554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.817529    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9" event={"ID":"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0","Type":"ContainerStarted","Data":"b77302fcba3cafa3033a0760926bce7a495265cc1900cad601737f16f09707da"}
Nov 02 17:36:59.817688 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.817620    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:36:59.835950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.835773    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87" podStartSLOduration=1.242684462 podStartE2EDuration="4.835755462s" totalImagesPullingTime="3.593071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:59.835507442 +0000 UTC m=+232.414916830" watchObservedRunningTime="2026-11-02 17:36:59.835755462 +0000 UTC m=+232.415164828"
Nov 02 17:36:59.857152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:36:59.857086    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9" podStartSLOduration=1.777478033 podStartE2EDuration="4.857067454s" totalImagesPullingTime="3.079589421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:59.855386179 +0000 UTC m=+232.434795565" watchObservedRunningTime="2026-11-02 17:36:59.857067454 +0000 UTC m=+232.436476842"
Nov 02 17:37:00.822331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:00.822287    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4sbhs" podUID="ec790129-59f9-4030-b260-23b2f7fcfc45" containerName="authorino" containerID="cri-o://043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f" gracePeriod=30
Nov 02 17:37:01.066237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.066209    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:37:01.157579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.157497    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4\" (UniqueName: \"kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4\") pod \"ec790129-59f9-4030-b260-23b2f7fcfc45\" (UID: \"ec790129-59f9-4030-b260-23b2f7fcfc45\") "
Nov 02 17:37:01.159762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.159736    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4" pod "ec790129-59f9-4030-b260-23b2f7fcfc45" (UID: "ec790129-59f9-4030-b260-23b2f7fcfc45"). InnerVolumeSpecName "kube-api-access-j9pc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:37:01.259028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.258989    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pc4\" (UniqueName: \"kubernetes.io/projected/ec790129-59f9-4030-b260-23b2f7fcfc45-kube-api-access-j9pc4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:37:01.827822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.827727    5042 generic.go:356] "Generic (PLEG): container finished" podID="ec790129-59f9-4030-b260-23b2f7fcfc45" containerID="043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f" exitCode=0
Nov 02 17:37:01.827822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.827772    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4sbhs"
Nov 02 17:37:01.827822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.827777    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4sbhs" event={"ID":"ec790129-59f9-4030-b260-23b2f7fcfc45","Type":"ContainerDied","Data":"043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f"}
Nov 02 17:37:01.828394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.827866    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4sbhs" event={"ID":"ec790129-59f9-4030-b260-23b2f7fcfc45","Type":"ContainerDied","Data":"fae7f2dcdc2b5b19f9cfa4c5685e4263531b52e35e563d06a4c833e0904696af"}
Nov 02 17:37:01.828394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.827891    5042 scope.go:122] "RemoveContainer" containerID="043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f"
Nov 02 17:37:01.837178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.837160    5042 scope.go:122] "RemoveContainer" containerID="043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f"
Nov 02 17:37:01.837464 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:37:01.837442    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f\": container with ID starting with 043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f not found: ID does not exist" containerID="043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f"
Nov 02 17:37:01.837507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.837477    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f"} err="failed to get container status \"043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f\": rpc error: code = NotFound desc = could not find container \"043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f\": container with ID starting with 043cd07d900af85c9f12d8e5136b0632bfb3cac0db0a75c2e69b4bdab956b95f not found: ID does not exist"
Nov 02 17:37:01.845345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.845321    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4sbhs"]
Nov 02 17:37:01.847985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:01.847964    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4sbhs"]
Nov 02 17:37:02.750828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:02.750801    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 17:37:03.541804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:03.541772    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec790129-59f9-4030-b260-23b2f7fcfc45" path="/var/lib/kubelet/pods/ec790129-59f9-4030-b260-23b2f7fcfc45/volumes"
Nov 02 17:37:10.827518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:10.827486    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6d9b8445f8-mbx87"
Nov 02 17:37:18.537733 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:37:18.537693    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:37:20.823585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:37:20.823551    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 17:37:48.538150 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:37:48.538110    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:38:07.556974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:07.556943    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:38:07.558894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:07.558872    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:38:07.559986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:07.559964    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:38:07.561991 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:07.561971    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:38:18.537927 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:38:18.537821    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:38:26.762383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.762343    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fsdxw"]
Nov 02 17:38:26.762781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.762738    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ec790129-59f9-4030-b260-23b2f7fcfc45" containerName="authorino"
Nov 02 17:38:26.762781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.762750    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec790129-59f9-4030-b260-23b2f7fcfc45" containerName="authorino"
Nov 02 17:38:26.762896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.762834    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec790129-59f9-4030-b260-23b2f7fcfc45" containerName="authorino"
Nov 02 17:38:26.766073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.766056    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.771569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.771540    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qjnd2\""
Nov 02 17:38:26.771569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.771549    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 17:38:26.772592 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.772568    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fsdxw"]
Nov 02 17:38:26.857387 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.857357    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6r8w\" (UniqueName: \"kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w\") pod \"authorino-6889cbd8c4-fsdxw\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") " pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.857574 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.857419    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert\") pod \"authorino-6889cbd8c4-fsdxw\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") " pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.958042 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.958007    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert\") pod \"authorino-6889cbd8c4-fsdxw\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") " pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.958208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.958101    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6r8w\" (UniqueName: \"kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w\") pod \"authorino-6889cbd8c4-fsdxw\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") " pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.960530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.960502    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert\") pod \"authorino-6889cbd8c4-fsdxw\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") " pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.969908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.969884    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6r8w\" (UniqueName: \"kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w\") pod \"authorino-6889cbd8c4-fsdxw\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") " pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:26.989314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.989288    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fsdxw"]
Nov 02 17:38:26.989507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:26.989495    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:27.272297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:27.272275    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fsdxw"]
Nov 02 17:38:27.274629 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:38:27.274596    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d708cc4_6624_4a90_b670_102e0b623541.slice/crio-ca310a41ea7dd934cd58c0b4156b3a9351089f2a4697a7a37510cf424c964c17 WatchSource:0}: Error finding container ca310a41ea7dd934cd58c0b4156b3a9351089f2a4697a7a37510cf424c964c17: Status 404 returned error can't find the container with id ca310a41ea7dd934cd58c0b4156b3a9351089f2a4697a7a37510cf424c964c17
Nov 02 17:38:27.276201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:27.276181    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:38:28.254989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.254933    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw" event={"ID":"7d708cc4-6624-4a90-b670-102e0b623541","Type":"ContainerStarted","Data":"b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940"}
Nov 02 17:38:28.254989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.254994    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw" event={"ID":"7d708cc4-6624-4a90-b670-102e0b623541","Type":"ContainerStarted","Data":"ca310a41ea7dd934cd58c0b4156b3a9351089f2a4697a7a37510cf424c964c17"}
Nov 02 17:38:28.255526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.255006    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw" podUID="7d708cc4-6624-4a90-b670-102e0b623541" containerName="authorino" containerID="cri-o://b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940" gracePeriod=30
Nov 02 17:38:28.276328 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.276063    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw" podStartSLOduration=1.39564204 podStartE2EDuration="2.276042438s" totalImagesPullingTime="880.400398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:28.272114724 +0000 UTC m=+320.851524111" watchObservedRunningTime="2026-11-02 17:38:28.276042438 +0000 UTC m=+320.855451825"
Nov 02 17:38:28.502073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.502042    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:28.674131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.674075    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert\" (UniqueName: \"kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert\") pod \"7d708cc4-6624-4a90-b670-102e0b623541\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") "
Nov 02 17:38:28.674338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.674157    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w\" (UniqueName: \"kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w\") pod \"7d708cc4-6624-4a90-b670-102e0b623541\" (UID: \"7d708cc4-6624-4a90-b670-102e0b623541\") "
Nov 02 17:38:28.676333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.676304    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w" pod "7d708cc4-6624-4a90-b670-102e0b623541" (UID: "7d708cc4-6624-4a90-b670-102e0b623541"). InnerVolumeSpecName "kube-api-access-s6r8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:38:28.685783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.685722    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert" pod "7d708cc4-6624-4a90-b670-102e0b623541" (UID: "7d708cc4-6624-4a90-b670-102e0b623541"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:38:28.775434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.775399    5042 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d708cc4-6624-4a90-b670-102e0b623541-tls-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:38:28.775434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:28.775432    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6r8w\" (UniqueName: \"kubernetes.io/projected/7d708cc4-6624-4a90-b670-102e0b623541-kube-api-access-s6r8w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:38:29.261311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.261275    5042 generic.go:356] "Generic (PLEG): container finished" podID="7d708cc4-6624-4a90-b670-102e0b623541" containerID="b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940" exitCode=0
Nov 02 17:38:29.261731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.261333    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw"
Nov 02 17:38:29.261731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.261364    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw" event={"ID":"7d708cc4-6624-4a90-b670-102e0b623541","Type":"ContainerDied","Data":"b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940"}
Nov 02 17:38:29.261731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.261426    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-fsdxw" event={"ID":"7d708cc4-6624-4a90-b670-102e0b623541","Type":"ContainerDied","Data":"ca310a41ea7dd934cd58c0b4156b3a9351089f2a4697a7a37510cf424c964c17"}
Nov 02 17:38:29.261731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.261447    5042 scope.go:122] "RemoveContainer" containerID="b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940"
Nov 02 17:38:29.270951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.270823    5042 scope.go:122] "RemoveContainer" containerID="b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940"
Nov 02 17:38:29.271241 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:38:29.271225    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940\": container with ID starting with b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940 not found: ID does not exist" containerID="b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940"
Nov 02 17:38:29.271290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.271250    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940"} err="failed to get container status \"b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940\": rpc error: code = NotFound desc = could not find container \"b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940\": container with ID starting with b25193d96735b9a79996a708737318fc9f2e8c157bce6904984ce4eef6b71940 not found: ID does not exist"
Nov 02 17:38:29.283281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.283228    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fsdxw"]
Nov 02 17:38:29.284787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.284767    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-fsdxw"]
Nov 02 17:38:29.541529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:38:29.541493    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d708cc4-6624-4a90-b670-102e0b623541" path="/var/lib/kubelet/pods/7d708cc4-6624-4a90-b670-102e0b623541/volumes"
Nov 02 17:38:48.537755 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:38:48.537702    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:39:18.537062 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:39:18.537030    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:39:48.537501 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:39:48.537422    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:40:03.294524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.294487    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"]
Nov 02 17:40:03.295646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.295620    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7d708cc4-6624-4a90-b670-102e0b623541" containerName="authorino"
Nov 02 17:40:03.295646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.295645    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d708cc4-6624-4a90-b670-102e0b623541" containerName="authorino"
Nov 02 17:40:03.295942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.295794    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d708cc4-6624-4a90-b670-102e0b623541" containerName="authorino"
Nov 02 17:40:03.299361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.299341    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.307829 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.307810    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 17:40:03.307985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.307926    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 17:40:03.308085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.308007    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 02 17:40:03.308162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.308094    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dnkjz\""
Nov 02 17:40:03.310699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.310677    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"]
Nov 02 17:40:03.425679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.425643    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpjt\" (UniqueName: \"kubernetes.io/projected/9f804eed-b3bb-4531-81b5-6060fd90682b-kube-api-access-4fpjt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.425856 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.425699    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.425907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.425826    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.425943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.425919    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.425976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.425952    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f804eed-b3bb-4531-81b5-6060fd90682b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.426025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.425978    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527140    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527189    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f804eed-b3bb-4531-81b5-6060fd90682b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527224    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527263    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpjt\" (UniqueName: \"kubernetes.io/projected/9f804eed-b3bb-4531-81b5-6060fd90682b-kube-api-access-4fpjt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527291    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527308    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527666    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.527692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.527683    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.528362 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.528331    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.541805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.541774    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f804eed-b3bb-4531-81b5-6060fd90682b-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.542461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.542439    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f804eed-b3bb-4531-81b5-6060fd90682b-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.544807 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.544747    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpjt\" (UniqueName: \"kubernetes.io/projected/9f804eed-b3bb-4531-81b5-6060fd90682b-kube-api-access-4fpjt\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4pb5f\" (UID: \"9f804eed-b3bb-4531-81b5-6060fd90682b\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.620870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.620805    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:03.905235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:03.905210    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"]
Nov 02 17:40:03.907271 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:03.907237    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f804eed_b3bb_4531_81b5_6060fd90682b.slice/crio-a70e6698c2e29f33c9a28433d44a5a6e1d4fa6f55b2c23cc247f661c107e52a3 WatchSource:0}: Error finding container a70e6698c2e29f33c9a28433d44a5a6e1d4fa6f55b2c23cc247f661c107e52a3: Status 404 returned error can't find the container with id a70e6698c2e29f33c9a28433d44a5a6e1d4fa6f55b2c23cc247f661c107e52a3
Nov 02 17:40:04.754515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:04.754477    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f" event={"ID":"9f804eed-b3bb-4531-81b5-6060fd90682b","Type":"ContainerStarted","Data":"a70e6698c2e29f33c9a28433d44a5a6e1d4fa6f55b2c23cc247f661c107e52a3"}
Nov 02 17:40:05.163365 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.163290    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"]
Nov 02 17:40:05.166953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.166931    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.181133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.177966    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 02 17:40:05.181133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.181025    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"]
Nov 02 17:40:05.244893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.244832    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.245026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.244957    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.245026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.244991    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ttn\" (UniqueName: \"kubernetes.io/projected/0b474fd7-9730-4acd-be17-6e9ae01a1e44-kube-api-access-p9ttn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.245150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.245118    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.245207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.245178    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b474fd7-9730-4acd-be17-6e9ae01a1e44-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.245248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.245230    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.346588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.346548    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.346588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.346590    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b474fd7-9730-4acd-be17-6e9ae01a1e44-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.346875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.346618    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.346875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.346644    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.346875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.346687    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.346875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.346718    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ttn\" (UniqueName: \"kubernetes.io/projected/0b474fd7-9730-4acd-be17-6e9ae01a1e44-kube-api-access-p9ttn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.347119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.347079    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.347159 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.347136    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.347202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.347181    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.348988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.348965    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b474fd7-9730-4acd-be17-6e9ae01a1e44-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.349251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.349234    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b474fd7-9730-4acd-be17-6e9ae01a1e44-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.356146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.356120    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ttn\" (UniqueName: \"kubernetes.io/projected/0b474fd7-9730-4acd-be17-6e9ae01a1e44-kube-api-access-p9ttn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd\" (UID: \"0b474fd7-9730-4acd-be17-6e9ae01a1e44\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.487745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.487703    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:05.761421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.761326    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f" event={"ID":"9f804eed-b3bb-4531-81b5-6060fd90682b","Type":"ContainerStarted","Data":"2d1513eab19dd26ab1d4e8212b9c406a2050cc55858129d80a43d9fae7c60091"}
Nov 02 17:40:05.770083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.770060    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"]
Nov 02 17:40:05.771412 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:05.771389    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b474fd7_9730_4acd_be17_6e9ae01a1e44.slice/crio-509fe5a6505f20ccd4c0dc2ab325676425a94963f86f421ebc2f9ebd9348f3bf WatchSource:0}: Error finding container 509fe5a6505f20ccd4c0dc2ab325676425a94963f86f421ebc2f9ebd9348f3bf: Status 404 returned error can't find the container with id 509fe5a6505f20ccd4c0dc2ab325676425a94963f86f421ebc2f9ebd9348f3bf
Nov 02 17:40:05.783731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.783685    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f" podStartSLOduration=1.4913243299999999 podStartE2EDuration="2.783669929s" totalImagesPullingTime="1.292345599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:05.782443674 +0000 UTC m=+418.361853059" watchObservedRunningTime="2026-11-02 17:40:05.783669929 +0000 UTC m=+418.363079314"
Nov 02 17:40:05.958087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.958052    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c8zlh"]
Nov 02 17:40:05.962918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.962893    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:05.968434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.968405    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 02 17:40:05.970656 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:05.970623    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c8zlh"]
Nov 02 17:40:06.055584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.055500    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-home\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.055584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.055547    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.055792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.055633    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.055792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.055667    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfbf\" (UniqueName: \"kubernetes.io/projected/23fa0458-aa25-47cf-9b62-00385b8646fd-kube-api-access-vsfbf\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.055792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.055706    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa0458-aa25-47cf-9b62-00385b8646fd-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.055919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.055792    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156286    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156330    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfbf\" (UniqueName: \"kubernetes.io/projected/23fa0458-aa25-47cf-9b62-00385b8646fd-kube-api-access-vsfbf\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156375    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa0458-aa25-47cf-9b62-00385b8646fd-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156458    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156600    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-home\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156645    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.156816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.156792    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.157024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.157003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.157333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.157069    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-home\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.159048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.159027    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23fa0458-aa25-47cf-9b62-00385b8646fd-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.159468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.159448    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23fa0458-aa25-47cf-9b62-00385b8646fd-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.165903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.165876    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfbf\" (UniqueName: \"kubernetes.io/projected/23fa0458-aa25-47cf-9b62-00385b8646fd-kube-api-access-vsfbf\") pod \"echo-simulator-kserve-64b6f8f96c-c8zlh\" (UID: \"23fa0458-aa25-47cf-9b62-00385b8646fd\") " pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.278484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.278448    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:06.558431 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.558402    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-c8zlh"]
Nov 02 17:40:06.560252 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:06.560218    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23fa0458_aa25_47cf_9b62_00385b8646fd.slice/crio-bdc2ae23d0cf6094b0e72f5bc69c57d9652e070e405f08de2e1b9f5bade84787 WatchSource:0}: Error finding container bdc2ae23d0cf6094b0e72f5bc69c57d9652e070e405f08de2e1b9f5bade84787: Status 404 returned error can't find the container with id bdc2ae23d0cf6094b0e72f5bc69c57d9652e070e405f08de2e1b9f5bade84787
Nov 02 17:40:06.622015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.621928    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:06.768669 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.768634    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd" event={"ID":"0b474fd7-9730-4acd-be17-6e9ae01a1e44","Type":"ContainerStarted","Data":"353981ee3965cfb627836a5a8849253219528c7385c18456d23393003475756b"}
Nov 02 17:40:06.769239 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.768696    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd" event={"ID":"0b474fd7-9730-4acd-be17-6e9ae01a1e44","Type":"ContainerStarted","Data":"509fe5a6505f20ccd4c0dc2ab325676425a94963f86f421ebc2f9ebd9348f3bf"}
Nov 02 17:40:06.769971 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.769941    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh" event={"ID":"23fa0458-aa25-47cf-9b62-00385b8646fd","Type":"ContainerStarted","Data":"bdc2ae23d0cf6094b0e72f5bc69c57d9652e070e405f08de2e1b9f5bade84787"}
Nov 02 17:40:06.792277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:06.792228    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd" podStartSLOduration=1.49740823 podStartE2EDuration="1.792216136s" totalImagesPullingTime="294.807906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:06.790264257 +0000 UTC m=+419.369673645" watchObservedRunningTime="2026-11-02 17:40:06.792216136 +0000 UTC m=+419.371625521"
Nov 02 17:40:07.488285 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:07.488246    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:08.148862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.148812    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"]
Nov 02 17:40:08.152907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.152885    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.159862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.159586    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 17:40:08.162453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.162208    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"]
Nov 02 17:40:08.278999 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.278943    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.278999 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.278983    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.279256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.279009    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2803f60a-6320-44ec-8961-4a6f026d6a45-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.279256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.279125    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.279337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.279269    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxnn\" (UniqueName: \"kubernetes.io/projected/2803f60a-6320-44ec-8961-4a6f026d6a45-kube-api-access-pjxnn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.279410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.279390    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380187    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380308 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380220    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380308 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380238    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2803f60a-6320-44ec-8961-4a6f026d6a45-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380359    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380486 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380461    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxnn\" (UniqueName: \"kubernetes.io/projected/2803f60a-6320-44ec-8961-4a6f026d6a45-kube-api-access-pjxnn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380549    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380622    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380643    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.380886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.380712    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.382733 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.382713    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2803f60a-6320-44ec-8961-4a6f026d6a45-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.382824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.382803    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2803f60a-6320-44ec-8961-4a6f026d6a45-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.393790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.392287    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxnn\" (UniqueName: \"kubernetes.io/projected/2803f60a-6320-44ec-8961-4a6f026d6a45-kube-api-access-pjxnn\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk\" (UID: \"2803f60a-6320-44ec-8961-4a6f026d6a45\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.477361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.477320    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:08.754517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.754494    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"]
Nov 02 17:40:08.756609 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:08.756581    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2803f60a_6320_44ec_8961_4a6f026d6a45.slice/crio-d7c27bf1fafb5d355cc94a236b9d5c03b308138212a1b6ff15238de7ed90c913 WatchSource:0}: Error finding container d7c27bf1fafb5d355cc94a236b9d5c03b308138212a1b6ff15238de7ed90c913: Status 404 returned error can't find the container with id d7c27bf1fafb5d355cc94a236b9d5c03b308138212a1b6ff15238de7ed90c913
Nov 02 17:40:08.788599 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.788558    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" event={"ID":"2803f60a-6320-44ec-8961-4a6f026d6a45","Type":"ContainerStarted","Data":"d7c27bf1fafb5d355cc94a236b9d5c03b308138212a1b6ff15238de7ed90c913"}
Nov 02 17:40:08.790496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.790466    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh" event={"ID":"23fa0458-aa25-47cf-9b62-00385b8646fd","Type":"ContainerStarted","Data":"f499ce3efbba4eaf4319dbfcfd8a56f217fb449ae54542f07dfebc31b2aaac36"}
Nov 02 17:40:08.812049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:08.812002    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh" podStartSLOduration=2.024080332 podStartE2EDuration="3.811988874s" totalImagesPullingTime="1.787908542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:08.81147294 +0000 UTC m=+421.390882337" watchObservedRunningTime="2026-11-02 17:40:08.811988874 +0000 UTC m=+421.391398257"
Nov 02 17:40:09.279037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:09.279000    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:09.798213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:09.798170    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" event={"ID":"2803f60a-6320-44ec-8961-4a6f026d6a45","Type":"ContainerStarted","Data":"26c20ae68a6dfbf71be8240a9f68922685ac36bf4950246badd82519cc496628"}
Nov 02 17:40:09.818215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:09.818161    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podStartSLOduration=1.524467351 podStartE2EDuration="1.818147873s" totalImagesPullingTime="293.680522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:09.81624323 +0000 UTC m=+422.395652619" watchObservedRunningTime="2026-11-02 17:40:09.818147873 +0000 UTC m=+422.397557258"
Nov 02 17:40:10.478146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:10.478099    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:13.621725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:13.621686    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:13.634494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:13.634464    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:13.834636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:13.834604    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f"
Nov 02 17:40:15.488221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:15.488188    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:15.500641 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:15.500613    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:15.840307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:15.840232    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd"
Nov 02 17:40:16.279032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:16.278999    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:16.291573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:16.291544    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:16.844685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:16.844656    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh"
Nov 02 17:40:18.478196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:18.478155    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:18.490949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:18.490918    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:18.537103 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:40:18.537067    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:40:18.856182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:18.856102    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:40:21.696457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.696417    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fb64bd955-5m6ww"]
Nov 02 17:40:21.700664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.700633    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.708745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.708714    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fb64bd955-5m6ww"]
Nov 02 17:40:21.708957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.708910    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qjnd2\""
Nov 02 17:40:21.709039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.709004    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 17:40:21.809480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.809446    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdjt\" (UniqueName: \"kubernetes.io/projected/25e41b3c-1fd8-422b-aab6-a660c5b76e4e-kube-api-access-skdjt\") pod \"authorino-5fb64bd955-5m6ww\" (UID: \"25e41b3c-1fd8-422b-aab6-a660c5b76e4e\") " pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.809677 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.809497    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25e41b3c-1fd8-422b-aab6-a660c5b76e4e-tls-cert\") pod \"authorino-5fb64bd955-5m6ww\" (UID: \"25e41b3c-1fd8-422b-aab6-a660c5b76e4e\") " pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.910348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.910315    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdjt\" (UniqueName: \"kubernetes.io/projected/25e41b3c-1fd8-422b-aab6-a660c5b76e4e-kube-api-access-skdjt\") pod \"authorino-5fb64bd955-5m6ww\" (UID: \"25e41b3c-1fd8-422b-aab6-a660c5b76e4e\") " pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.910548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.910378    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25e41b3c-1fd8-422b-aab6-a660c5b76e4e-tls-cert\") pod \"authorino-5fb64bd955-5m6ww\" (UID: \"25e41b3c-1fd8-422b-aab6-a660c5b76e4e\") " pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.913165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.913141    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/25e41b3c-1fd8-422b-aab6-a660c5b76e4e-tls-cert\") pod \"authorino-5fb64bd955-5m6ww\" (UID: \"25e41b3c-1fd8-422b-aab6-a660c5b76e4e\") " pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.922587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.922563    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdjt\" (UniqueName: \"kubernetes.io/projected/25e41b3c-1fd8-422b-aab6-a660c5b76e4e-kube-api-access-skdjt\") pod \"authorino-5fb64bd955-5m6ww\" (UID: \"25e41b3c-1fd8-422b-aab6-a660c5b76e4e\") " pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:21.940093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.940052    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"]
Nov 02 17:40:21.944132 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.944113    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:21.951735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.951427    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qbv6v\""
Nov 02 17:40:21.957386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:21.957363    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"]
Nov 02 17:40:22.011569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011522    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.011750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011579    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nvm\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.011750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011728    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.011873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011798    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.011926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011884    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.011926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011921    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.012010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011944    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.012010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.011981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.012010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.012010    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.015432 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.015415    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fb64bd955-5m6ww"
Nov 02 17:40:22.113557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113524    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113564    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113601    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113647    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113681    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113731    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113755    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.113823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113798    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.114157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.113870    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nvm\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.114231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.114202    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.114385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.114365    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.114452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.114391    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.114541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.114453    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.114679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.114660    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.116182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.116158    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.116392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.116373    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.123029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.122999    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nvm\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.123333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.123311    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-m6g9w\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.260274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.260194    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:22.292070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.292038    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fb64bd955-5m6ww"]
Nov 02 17:40:22.294359 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:22.294315    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25e41b3c_1fd8_422b_aab6_a660c5b76e4e.slice/crio-5cc49d03af56f8a1d7d15812176ea13e23e30f261e281c493971b1ce0b8bc1c4 WatchSource:0}: Error finding container 5cc49d03af56f8a1d7d15812176ea13e23e30f261e281c493971b1ce0b8bc1c4: Status 404 returned error can't find the container with id 5cc49d03af56f8a1d7d15812176ea13e23e30f261e281c493971b1ce0b8bc1c4
Nov 02 17:40:22.548668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.548643    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"]
Nov 02 17:40:22.551705 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:22.551652    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344a2e2e_0404_446b_844d_47e6baa07dfb.slice/crio-7cf0730c78411dc28cae005d8b9ec1d217ae92c107850505124c3b36581a1881 WatchSource:0}: Error finding container 7cf0730c78411dc28cae005d8b9ec1d217ae92c107850505124c3b36581a1881: Status 404 returned error can't find the container with id 7cf0730c78411dc28cae005d8b9ec1d217ae92c107850505124c3b36581a1881
Nov 02 17:40:22.555386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.555351    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:22.555477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.555433    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:22.555521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.555472    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:22.867293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.867201    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fb64bd955-5m6ww" event={"ID":"25e41b3c-1fd8-422b-aab6-a660c5b76e4e","Type":"ContainerStarted","Data":"5cc49d03af56f8a1d7d15812176ea13e23e30f261e281c493971b1ce0b8bc1c4"}
Nov 02 17:40:22.869040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.869010    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" event={"ID":"344a2e2e-0404-446b-844d-47e6baa07dfb","Type":"ContainerStarted","Data":"19dd6d3721e32d07d8d6dfdf8b6575ea295770067860d2bb72781a73e7eaa053"}
Nov 02 17:40:22.869177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.869133    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" event={"ID":"344a2e2e-0404-446b-844d-47e6baa07dfb","Type":"ContainerStarted","Data":"7cf0730c78411dc28cae005d8b9ec1d217ae92c107850505124c3b36581a1881"}
Nov 02 17:40:22.891897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:22.891828    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podStartSLOduration=1.891811755 podStartE2EDuration="1.891811755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:22.8881166 +0000 UTC m=+435.467525982" watchObservedRunningTime="2026-11-02 17:40:22.891811755 +0000 UTC m=+435.471221140"
Nov 02 17:40:23.261238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:23.261204    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:23.262794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:23.262767    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:23.262922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:23.262814    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:23.903809 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:23.903768    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fb64bd955-5m6ww" event={"ID":"25e41b3c-1fd8-422b-aab6-a660c5b76e4e","Type":"ContainerStarted","Data":"967e810faada77192ae432773723c111056d41586db3a2ad116e742e4a8e89bb"}
Nov 02 17:40:23.925581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:23.925518    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fb64bd955-5m6ww" podStartSLOduration=1.8573412519999999 podStartE2EDuration="2.92549944s" totalImagesPullingTime="1.068158188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:23.921312179 +0000 UTC m=+436.500721566" watchObservedRunningTime="2026-11-02 17:40:23.92549944 +0000 UTC m=+436.504908825"
Nov 02 17:40:24.261458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:24.261424    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:24.261633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:24.261483    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:25.261033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:25.260993    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:25.261431 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:25.261061    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:26.260764 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:26.260727    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:26.260946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:26.260794    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:27.260558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:27.260521    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:27.260974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:27.260597    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:28.261543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:28.261501    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:28.261947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:28.261558    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:29.261116 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:29.261072    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Nov 02 17:40:29.261297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:29.261134    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Nov 02 17:40:30.963776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:30.963738    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:30.964442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:30.964374    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:30.965116 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:30.965086    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:40:36.957106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:36.957069    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"]
Nov 02 17:40:36.962944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:36.962921    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:36.975658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:36.975631    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"]
Nov 02 17:40:37.019326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019297    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019351    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019419    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019465    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019490    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019514    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019592    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g622r\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019631    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.019652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.019647    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120207    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120251    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120286    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120334    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120358    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120371    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120437    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120463    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g622r\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120744    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.120995 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.120833    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.121100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.121050    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.121158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.121092    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.121358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.121334    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.122686 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.122671    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.122987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.122970    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.134700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.134676    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.134876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.134855    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g622r\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r\") pod \"maas-default-gateway-openshift-default-65c448b547-b7gw6\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.277979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.277892    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:37.558777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.558753    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"]
Nov 02 17:40:37.560804 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:37.560765    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b1d399_8970_4edf_a253_6a032bc6e9c9.slice/crio-d3c396a71d21643dcaa45b9afc0227d4bab0a751d77f9fe6d36cac333ce21f88 WatchSource:0}: Error finding container d3c396a71d21643dcaa45b9afc0227d4bab0a751d77f9fe6d36cac333ce21f88: Status 404 returned error can't find the container with id d3c396a71d21643dcaa45b9afc0227d4bab0a751d77f9fe6d36cac333ce21f88
Nov 02 17:40:37.563073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.563046    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:37.563156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.563107    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:37.563156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:37.563135    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:38.017270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:38.017224    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" event={"ID":"76b1d399-8970-4edf-a253-6a032bc6e9c9","Type":"ContainerStarted","Data":"b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40"}
Nov 02 17:40:38.017655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:38.017350    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" event={"ID":"76b1d399-8970-4edf-a253-6a032bc6e9c9","Type":"ContainerStarted","Data":"d3c396a71d21643dcaa45b9afc0227d4bab0a751d77f9fe6d36cac333ce21f88"}
Nov 02 17:40:38.043526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:38.043475    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podStartSLOduration=2.043462376 podStartE2EDuration="2.043462376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:38.041498402 +0000 UTC m=+450.620907789" watchObservedRunningTime="2026-11-02 17:40:38.043462376 +0000 UTC m=+450.622871811"
Nov 02 17:40:38.291921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:38.279227    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:38.291921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:38.287617    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:38.291921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:38.287682    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:39.290867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:39.286263    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:39.290867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:39.286330    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:40.293864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:40.291300    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:40.293864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:40.291378    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:41.293865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:41.291320    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:41.293865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:41.291383    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:42.280867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:42.279379    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:42.280867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:42.279457    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:43.279118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:43.278603    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:43.279118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:43.278670    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:44.278939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:44.278909    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Nov 02 17:40:44.279120 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:44.278974    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Nov 02 17:40:45.571048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:45.571011    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:45.571792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:45.571768    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:45.573163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:45.573139    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:40:48.537143 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:40:48.537101    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:40:59.232109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.232075    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"]
Nov 02 17:40:59.236250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.236229    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.245916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.245893    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-38f4eea3-gw-openshift-default-dockercfg-vbg2k\""
Nov 02 17:40:59.253353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.253324    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"]
Nov 02 17:40:59.322785 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.322734    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.322785 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.322788    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8kg\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.322809    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.322921    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323365 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.323242    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323415 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.323360    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.323419    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.323453    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.323614 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.323596    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.424994    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425045    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8kg\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425175    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425209    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425239    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425281    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425334    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425379    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425435    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425573    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425748 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425632    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425906    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.425990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.425909    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.426334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.426309    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.428178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.428159    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.428321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.428303    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.436073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.436041    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.436268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.436245    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8kg\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg\") pod \"e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") " pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.558265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.558234    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:40:59.844409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.844383    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"]
Nov 02 17:40:59.846471 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:40:59.846438    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d74498_bf13_4db1_be63_6ba16d9c93dd.slice/crio-4a6bcf14a32571f7df8469a55e2991c776e21f8d3bbccc48d97b461be0c8c04e WatchSource:0}: Error finding container 4a6bcf14a32571f7df8469a55e2991c776e21f8d3bbccc48d97b461be0c8c04e: Status 404 returned error can't find the container with id 4a6bcf14a32571f7df8469a55e2991c776e21f8d3bbccc48d97b461be0c8c04e
Nov 02 17:40:59.849925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.849893    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:59.850026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.849980    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:59.850143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:40:59.850027    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:00.133360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:00.133267    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" event={"ID":"75d74498-bf13-4db1-be63-6ba16d9c93dd","Type":"ContainerStarted","Data":"d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"}
Nov 02 17:41:00.133360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:00.133321    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" event={"ID":"75d74498-bf13-4db1-be63-6ba16d9c93dd","Type":"ContainerStarted","Data":"4a6bcf14a32571f7df8469a55e2991c776e21f8d3bbccc48d97b461be0c8c04e"}
Nov 02 17:41:00.154438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:00.154378    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podStartSLOduration=1.154363606 podStartE2EDuration="1.154363606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:00.152928381 +0000 UTC m=+472.732337770" watchObservedRunningTime="2026-11-02 17:41:00.154363606 +0000 UTC m=+472.733772992"
Nov 02 17:41:00.558411 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:00.558374    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:00.563698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:00.563669    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:01.139099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:01.139069    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:01.140131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:01.140113    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:03.962704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:03.962670    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"]
Nov 02 17:41:03.966912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:03.966888    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:03.980086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:03.980055    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ad831efe-dockercfg-5nqbt\""
Nov 02 17:41:03.982044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:03.982012    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"]
Nov 02 17:41:04.053147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.053114    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad831efe\""
Nov 02 17:41:04.087526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.087486    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.087823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.087799    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nx5z\" (UniqueName: \"kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z\") pod \"payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.189295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.189250    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nx5z\" (UniqueName: \"kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z\") pod \"payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.189492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.189472    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.190461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.190436    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.198149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.198111    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nx5z\" (UniqueName: \"kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z\") pod \"payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.285381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.285285    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:04.578203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:04.578174    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"]
Nov 02 17:41:04.580672 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:04.580636    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cbc233_d835_4531_b5a4_cede4eeefeca.slice/crio-fc7c8c4c80ea7104b56c96edcae77e9a96da13f817ca3b687beb7d528d9ee371 WatchSource:0}: Error finding container fc7c8c4c80ea7104b56c96edcae77e9a96da13f817ca3b687beb7d528d9ee371: Status 404 returned error can't find the container with id fc7c8c4c80ea7104b56c96edcae77e9a96da13f817ca3b687beb7d528d9ee371
Nov 02 17:41:05.006302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.006231    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"]
Nov 02 17:41:05.011541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.011510    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.020396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.017622    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0ae9c41e-dockercfg-rmksk\""
Nov 02 17:41:05.020396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.017690    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ae9c41e\""
Nov 02 17:41:05.020396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.019578    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"]
Nov 02 17:41:05.109680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.108492    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xg2c\" (UniqueName: \"kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c\") pod \"payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") " pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.109680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.108616    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") " pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.165102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.165058    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p" event={"ID":"92cbc233-d835-4531-b5a4-cede4eeefeca","Type":"ContainerStarted","Data":"fc7c8c4c80ea7104b56c96edcae77e9a96da13f817ca3b687beb7d528d9ee371"}
Nov 02 17:41:05.210086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.210045    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") " pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.210258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.210173    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xg2c\" (UniqueName: \"kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c\") pod \"payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") " pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.215532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.215499    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") " pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.223474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.223442    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xg2c\" (UniqueName: \"kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c\") pod \"payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") " pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.330861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.330768    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:05.646022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:05.645989    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"]
Nov 02 17:41:06.198020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.197766    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9" event={"ID":"d8e3c225-09e7-4337-be11-9bd8e0b90be8","Type":"ContainerStarted","Data":"854d70583d832c1924e40837de819ebe12fb43edcbae0bf6c16bdffff315a5ab"}
Nov 02 17:41:06.716559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.716504    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"]
Nov 02 17:41:06.736869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.736642    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"]
Nov 02 17:41:06.737056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.736759    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.753237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.753190    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-55f9886f-openshift-default-dockercfg-xgsxd\""
Nov 02 17:41:06.844516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.841615    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.844919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.844829    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.845115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.845094    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.845233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.845217    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.845374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.845348    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsjp\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.845529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.845513    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.882871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.872403    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.882871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.872476    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.882871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.872521    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.973830    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.973935    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.973968    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.974088    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.974158    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.974194    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.974222    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.974251    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsjp\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.974291    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.975087    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.975375    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.975948    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.976182    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.978907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.976764    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:06.997658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:06.997146    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:07.007603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.007386    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:07.008325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.008273    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:07.010223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.010183    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsjp\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp\") pod \"e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") " pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:07.055644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.055535    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:07.416247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.415982    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"]
Nov 02 17:41:07.421732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.417713    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:07.421732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.417802    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:07.421732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:07.417865    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:08.278868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:08.257048    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" event={"ID":"a4ce0beb-a0e5-4929-98fb-5b39970c6c99","Type":"ContainerStarted","Data":"4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"}
Nov 02 17:41:08.278868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:08.257120    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" event={"ID":"a4ce0beb-a0e5-4929-98fb-5b39970c6c99","Type":"ContainerStarted","Data":"11b71d382b45b17ac471953125840dff9205227799c85fa8f75538820d7ebdc9"}
Nov 02 17:41:08.312204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:08.287816    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podStartSLOduration=2.287792048 podStartE2EDuration="2.287792048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:08.285463179 +0000 UTC m=+480.864872578" watchObservedRunningTime="2026-11-02 17:41:08.287792048 +0000 UTC m=+480.867201432"
Nov 02 17:41:09.055799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.055767    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:09.062368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.062334    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:09.269782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.269747    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p" event={"ID":"92cbc233-d835-4531-b5a4-cede4eeefeca","Type":"ContainerStarted","Data":"9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7"}
Nov 02 17:41:09.269959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.269813    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:09.269959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.269894    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:09.271063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.271037    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:09.308944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:09.294707    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p" podStartSLOduration=1.744277022 podStartE2EDuration="6.294687059s" totalImagesPullingTime="4.550410037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:09.291251768 +0000 UTC m=+481.870661179" watchObservedRunningTime="2026-11-02 17:41:09.294687059 +0000 UTC m=+481.874096447"
Nov 02 17:41:10.275751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:10.275708    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9" event={"ID":"d8e3c225-09e7-4337-be11-9bd8e0b90be8","Type":"ContainerStarted","Data":"2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec"}
Nov 02 17:41:10.299011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:10.298954    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9" podStartSLOduration=2.01268038 podStartE2EDuration="6.298938053s" totalImagesPullingTime="4.286257673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:10.297062456 +0000 UTC m=+482.876471842" watchObservedRunningTime="2026-11-02 17:41:10.298938053 +0000 UTC m=+482.878347439"
Nov 02 17:41:11.121708 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.121667    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"]
Nov 02 17:41:11.125650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.125625    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.144159 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.144126    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-55f9886f\""
Nov 02 17:41:11.144344 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.144279    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-55f9886f-dockercfg-s5cpp\""
Nov 02 17:41:11.145998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.145960    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"]
Nov 02 17:41:11.172330 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.172295    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.172514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.172468    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzfb\" (UniqueName: \"kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb\") pod \"payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.273667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.273623    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.273904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.273738    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzfb\" (UniqueName: \"kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb\") pod \"payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.274444 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.274420    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.283519 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.283487    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzfb\" (UniqueName: \"kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb\") pod \"payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.283984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.283910    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:11.459575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.459536    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:11.759019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:11.758992    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"]
Nov 02 17:41:11.760991 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:11.760957    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61396460_12ef_4e0a_a5c4_2aca8b73f37c.slice/crio-7021f21f24f892f68f49b64593305a308429a8939dfd618c54ac305ab84cb890 WatchSource:0}: Error finding container 7021f21f24f892f68f49b64593305a308429a8939dfd618c54ac305ab84cb890: Status 404 returned error can't find the container with id 7021f21f24f892f68f49b64593305a308429a8939dfd618c54ac305ab84cb890
Nov 02 17:41:12.326452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:12.326412    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k" event={"ID":"61396460-12ef-4e0a-a5c4-2aca8b73f37c","Type":"ContainerStarted","Data":"158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a"}
Nov 02 17:41:12.326988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:12.326474    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k" event={"ID":"61396460-12ef-4e0a-a5c4-2aca8b73f37c","Type":"ContainerStarted","Data":"7021f21f24f892f68f49b64593305a308429a8939dfd618c54ac305ab84cb890"}
Nov 02 17:41:12.326988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:12.326537    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:12.349959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:12.347571    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k" podStartSLOduration=1.347552551 podStartE2EDuration="1.347552551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:12.345896571 +0000 UTC m=+484.925305958" watchObservedRunningTime="2026-11-02 17:41:12.347552551 +0000 UTC m=+484.926961941"
Nov 02 17:41:14.339544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:14.339455    5042 generic.go:356] "Generic (PLEG): container finished" podID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerID="d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a" exitCode=0
Nov 02 17:41:14.339544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:14.339527    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" event={"ID":"75d74498-bf13-4db1-be63-6ba16d9c93dd","Type":"ContainerDied","Data":"d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"}
Nov 02 17:41:14.340100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:14.340085    5042 scope.go:122] "RemoveContainer" containerID="d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"
Nov 02 17:41:14.341624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:14.341594    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:14.341710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:14.341693    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:14.341768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:14.341752    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:15.352390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:15.352345    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" event={"ID":"75d74498-bf13-4db1-be63-6ba16d9c93dd","Type":"ContainerStarted","Data":"1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954"}
Nov 02 17:41:15.559827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:15.559787    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:15.560412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:15.560364    5042 patch_prober.go:29] interesting pod/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:41:15.560666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:15.560616    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:41:16.139977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.139940    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:16.559403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.559371    5042 patch_prober.go:29] interesting pod/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:41:16.559907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.559880    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:41:16.831916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.824626    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"]
Nov 02 17:41:16.840640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.840605    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"]
Nov 02 17:41:16.841010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.840991    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:16.848115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.848086    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:41:16.848394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.848380    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:41:16.848547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.848536    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:41:16.848707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.848696    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:41:16.958465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.958429    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:16.958643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.958547    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:16.958643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:16.958588    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dtx\" (UniqueName: \"kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.059575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.059539    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.059796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.059607    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.059796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.059633    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dtx\" (UniqueName: \"kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.060392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.060367    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.062332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.062309    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.069312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.069288    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dtx\" (UniqueName: \"kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx\") pod \"maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.229865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.218038    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:17.558971 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.558899    5042 patch_prober.go:29] interesting pod/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:41:17.559105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.558965    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:41:17.604273 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:17.603078    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"]
Nov 02 17:41:18.392865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:18.390637    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj" event={"ID":"065964b7-8560-4d1d-9e8a-32b5d72f946c","Type":"ContainerStarted","Data":"65b7ad0f9562794448a2cd67fe8c436fe830753e8c20fdb42ad3cc01dc4cc0d8"}
Nov 02 17:41:18.557390 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:18.557354    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:41:18.558787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:18.558759    5042 patch_prober.go:29] interesting pod/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:41:18.558938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:18.558816    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:41:19.511830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.511789    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"]
Nov 02 17:41:19.534438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.534407    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"]
Nov 02 17:41:19.534775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.534756    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.559383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.559350    5042 patch_prober.go:29] interesting pod/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:41:19.559553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.559412    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:41:19.634879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.634738    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzpg\" (UniqueName: \"kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.634879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.634807    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.635139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.634988    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.740267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.739493    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzpg\" (UniqueName: \"kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.740267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.739564    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.740267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.739718    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.740735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.740695    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.744529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.744497    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.749569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.749544    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzpg\" (UniqueName: \"kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg\") pod \"maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:19.855463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:19.855383    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:20.243580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:20.243160    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"]
Nov 02 17:41:20.246266 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:20.246233    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d52df2a_e352_4df2_9869_1979bef7dc17.slice/crio-a51da1a4a772b791cca5b746454cd8facea00729579b0bd33ee46db2862645ca WatchSource:0}: Error finding container a51da1a4a772b791cca5b746454cd8facea00729579b0bd33ee46db2862645ca: Status 404 returned error can't find the container with id a51da1a4a772b791cca5b746454cd8facea00729579b0bd33ee46db2862645ca
Nov 02 17:41:20.445978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:20.445935    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn" event={"ID":"7d52df2a-e352-4df2-9869-1979bef7dc17","Type":"ContainerStarted","Data":"a51da1a4a772b791cca5b746454cd8facea00729579b0bd33ee46db2862645ca"}
Nov 02 17:41:20.560353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:20.559821    5042 patch_prober.go:29] interesting pod/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:41:20.560353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:20.559907    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 02 17:41:21.303106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:21.303067    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:21.567776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:21.565495    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:21.567776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:21.567317    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:22.330727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.330695    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:22.485417 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.485328    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"]
Nov 02 17:41:22.496757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.496612    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.502621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.502586    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"]
Nov 02 17:41:22.523154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.522641    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.523154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.522902    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntj9\" (UniqueName: \"kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.524680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.523578    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.624598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.624558    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.624997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.624670    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntj9\" (UniqueName: \"kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.624997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.624723    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.626182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.626159    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.628655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.628618    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.644874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.644812    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntj9\" (UniqueName: \"kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9\") pod \"maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:22.827155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:22.827068    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:23.159406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:23.159365    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"]
Nov 02 17:41:23.161964 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:23.161937    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58f3bae_45b4_4c49_a7e1_e844bd9dfa6b.slice/crio-1ed0f4f25c1c315d0c1fef6227cc93968c34a59ee7c07045d0c703dd2b6d6c74 WatchSource:0}: Error finding container 1ed0f4f25c1c315d0c1fef6227cc93968c34a59ee7c07045d0c703dd2b6d6c74: Status 404 returned error can't find the container with id 1ed0f4f25c1c315d0c1fef6227cc93968c34a59ee7c07045d0c703dd2b6d6c74
Nov 02 17:41:23.478994 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:23.474883    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg" event={"ID":"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b","Type":"ContainerStarted","Data":"1ed0f4f25c1c315d0c1fef6227cc93968c34a59ee7c07045d0c703dd2b6d6c74"}
Nov 02 17:41:24.343756 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.343729    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:24.486467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.486426    5042 generic.go:356] "Generic (PLEG): container finished" podID="c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" containerID="892f80d7e36948497b7f6806eede06671cd69fabeae04f21c8e5755f685743ac" exitCode=0
Nov 02 17:41:24.486644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.486512    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg" event={"ID":"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b","Type":"ContainerDied","Data":"892f80d7e36948497b7f6806eede06671cd69fabeae04f21c8e5755f685743ac"}
Nov 02 17:41:24.488772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.488740    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn" event={"ID":"7d52df2a-e352-4df2-9869-1979bef7dc17","Type":"ContainerStarted","Data":"5443cf15779a25447c460d79917b782b26e3485baab1951efeea161b57a6e075"}
Nov 02 17:41:24.490776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.490747    5042 generic.go:356] "Generic (PLEG): container finished" podID="065964b7-8560-4d1d-9e8a-32b5d72f946c" containerID="bd2695947087fe83af782996d333fda0f06b5b4a6253fba1db30c21de65fd772" exitCode=0
Nov 02 17:41:24.490943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.490834    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj" event={"ID":"065964b7-8560-4d1d-9e8a-32b5d72f946c","Type":"ContainerDied","Data":"bd2695947087fe83af782996d333fda0f06b5b4a6253fba1db30c21de65fd772"}
Nov 02 17:41:24.525805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.525688    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn" podStartSLOduration=2.050786005 podStartE2EDuration="5.525666265s" totalImagesPullingTime="3.47488026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:24.522330677 +0000 UTC m=+497.101740245" watchObservedRunningTime="2026-11-02 17:41:24.525666265 +0000 UTC m=+497.105075653"
Nov 02 17:41:24.530625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.530534    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"]
Nov 02 17:41:24.534486 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.534466    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.543915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.542689    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"]
Nov 02 17:41:24.553079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.553038    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.553259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.553181    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnrn\" (UniqueName: \"kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.553467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.553444    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.654996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.654952    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.655180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.655031    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.655180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.655063    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnrn\" (UniqueName: \"kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.655855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.655801    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.657602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.657582    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.663769 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.663747    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnrn\" (UniqueName: \"kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn\") pod \"maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:24.857462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:24.857371    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:25.141865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.141801    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"]
Nov 02 17:41:25.144468 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:25.144435    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc82128_f84a_4563_9920_af57cf6e2c1c.slice/crio-2b9f2f6d84d8af24e3983fb3c3c435ae7aaff9b36b6b46d0e9e49256a0c65a11 WatchSource:0}: Error finding container 2b9f2f6d84d8af24e3983fb3c3c435ae7aaff9b36b6b46d0e9e49256a0c65a11: Status 404 returned error can't find the container with id 2b9f2f6d84d8af24e3983fb3c3c435ae7aaff9b36b6b46d0e9e49256a0c65a11
Nov 02 17:41:25.311047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.311006    5042 patch_prober.go:29] interesting pod/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:41:25.311224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.311077    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:41:25.499209 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.499171    5042 generic.go:356] "Generic (PLEG): container finished" podID="7d52df2a-e352-4df2-9869-1979bef7dc17" containerID="5443cf15779a25447c460d79917b782b26e3485baab1951efeea161b57a6e075" exitCode=0
Nov 02 17:41:25.499709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.499265    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn" event={"ID":"7d52df2a-e352-4df2-9869-1979bef7dc17","Type":"ContainerDied","Data":"5443cf15779a25447c460d79917b782b26e3485baab1951efeea161b57a6e075"}
Nov 02 17:41:25.501005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.500973    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c" event={"ID":"6dc82128-f84a-4563-9920-af57cf6e2c1c","Type":"ContainerStarted","Data":"2b9f2f6d84d8af24e3983fb3c3c435ae7aaff9b36b6b46d0e9e49256a0c65a11"}
Nov 02 17:41:25.698024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.697996    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:25.730909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.730832    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:25.769221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.769127    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca\") pod \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") "
Nov 02 17:41:25.769221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.769216    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx\" (UniqueName: \"kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx\") pod \"065964b7-8560-4d1d-9e8a-32b5d72f946c\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") "
Nov 02 17:41:25.769643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.769266    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca\") pod \"065964b7-8560-4d1d-9e8a-32b5d72f946c\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") "
Nov 02 17:41:25.769643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.769317    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9\" (UniqueName: \"kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9\") pod \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") "
Nov 02 17:41:25.769643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.769391    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca\") pod \"065964b7-8560-4d1d-9e8a-32b5d72f946c\" (UID: \"065964b7-8560-4d1d-9e8a-32b5d72f946c\") "
Nov 02 17:41:25.769643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.769438    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca\") pod \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\" (UID: \"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b\") "
Nov 02 17:41:25.770181 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.770148    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca" pod "065964b7-8560-4d1d-9e8a-32b5d72f946c" (UID: "065964b7-8560-4d1d-9e8a-32b5d72f946c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:25.770292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.770221    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca" pod "c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" (UID: "c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:25.772802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.772773    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca" pod "c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" (UID: "c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:25.773527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.773503    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx" pod "065964b7-8560-4d1d-9e8a-32b5d72f946c" (UID: "065964b7-8560-4d1d-9e8a-32b5d72f946c"). InnerVolumeSpecName "kube-api-access-44dtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:25.773743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.773716    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9" pod "c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" (UID: "c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b"). InnerVolumeSpecName "kube-api-access-gntj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:25.773954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.773932    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca" pod "065964b7-8560-4d1d-9e8a-32b5d72f946c" (UID: "065964b7-8560-4d1d-9e8a-32b5d72f946c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:25.871250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.871209    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:25.871250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.871246    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:25.871504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.871261    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:25.871504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.871283    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44dtx\" (UniqueName: \"kubernetes.io/projected/065964b7-8560-4d1d-9e8a-32b5d72f946c-kube-api-access-44dtx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:25.871504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.871297    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/065964b7-8560-4d1d-9e8a-32b5d72f946c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:25.871504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:25.871311    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntj9\" (UniqueName: \"kubernetes.io/projected/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b-kube-api-access-gntj9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:26.508355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.508321    5042 generic.go:356] "Generic (PLEG): container finished" podID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerID="4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591" exitCode=0
Nov 02 17:41:26.508879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.508398    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" event={"ID":"a4ce0beb-a0e5-4929-98fb-5b39970c6c99","Type":"ContainerDied","Data":"4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"}
Nov 02 17:41:26.508879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.508863    5042 scope.go:122] "RemoveContainer" containerID="4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"
Nov 02 17:41:26.510150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.510134    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"
Nov 02 17:41:26.510249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.510162    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj" event={"ID":"065964b7-8560-4d1d-9e8a-32b5d72f946c","Type":"ContainerDied","Data":"65b7ad0f9562794448a2cd67fe8c436fe830753e8c20fdb42ad3cc01dc4cc0d8"}
Nov 02 17:41:26.510249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.510203    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b7ad0f9562794448a2cd67fe8c436fe830753e8c20fdb42ad3cc01dc4cc0d8"
Nov 02 17:41:26.510249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.510195    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:26.510368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.510283    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:26.510368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.510323    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:26.511897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.511862    5042 generic.go:356] "Generic (PLEG): container finished" podID="6dc82128-f84a-4563-9920-af57cf6e2c1c" containerID="9ff97b7e40a65a28103e87081232c726e1c5866b290e58b5b6043cb59ccd3efa" exitCode=0
Nov 02 17:41:26.511897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.511885    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c" event={"ID":"6dc82128-f84a-4563-9920-af57cf6e2c1c","Type":"ContainerDied","Data":"9ff97b7e40a65a28103e87081232c726e1c5866b290e58b5b6043cb59ccd3efa"}
Nov 02 17:41:26.513321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.513303    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"
Nov 02 17:41:26.513429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.513328    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg" event={"ID":"c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b","Type":"ContainerDied","Data":"1ed0f4f25c1c315d0c1fef6227cc93968c34a59ee7c07045d0c703dd2b6d6c74"}
Nov 02 17:41:26.513429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.513366    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ed0f4f25c1c315d0c1fef6227cc93968c34a59ee7c07045d0c703dd2b6d6c74"
Nov 02 17:41:26.658632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.658603    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:26.690867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.679764    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca\") pod \"7d52df2a-e352-4df2-9869-1979bef7dc17\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") "
Nov 02 17:41:26.690867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.679891    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca\") pod \"7d52df2a-e352-4df2-9869-1979bef7dc17\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") "
Nov 02 17:41:26.690867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.680311    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca" pod "7d52df2a-e352-4df2-9869-1979bef7dc17" (UID: "7d52df2a-e352-4df2-9869-1979bef7dc17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:26.690867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.680742    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg\" (UniqueName: \"kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg\") pod \"7d52df2a-e352-4df2-9869-1979bef7dc17\" (UID: \"7d52df2a-e352-4df2-9869-1979bef7dc17\") "
Nov 02 17:41:26.690867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.687512    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:26.700300 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.700252    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg" pod "7d52df2a-e352-4df2-9869-1979bef7dc17" (UID: "7d52df2a-e352-4df2-9869-1979bef7dc17"). InnerVolumeSpecName "kube-api-access-ltzpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:26.703463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.703434    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca" pod "7d52df2a-e352-4df2-9869-1979bef7dc17" (UID: "7d52df2a-e352-4df2-9869-1979bef7dc17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:26.788779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.788695    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzpg\" (UniqueName: \"kubernetes.io/projected/7d52df2a-e352-4df2-9869-1979bef7dc17-kube-api-access-ltzpg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:26.788779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:26.788728    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d52df2a-e352-4df2-9869-1979bef7dc17-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:27.522391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.522357    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"
Nov 02 17:41:27.523070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.522362    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn" event={"ID":"7d52df2a-e352-4df2-9869-1979bef7dc17","Type":"ContainerDied","Data":"a51da1a4a772b791cca5b746454cd8facea00729579b0bd33ee46db2862645ca"}
Nov 02 17:41:27.523217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.523161    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a51da1a4a772b791cca5b746454cd8facea00729579b0bd33ee46db2862645ca"
Nov 02 17:41:27.525119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.525084    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" event={"ID":"a4ce0beb-a0e5-4929-98fb-5b39970c6c99","Type":"ContainerStarted","Data":"89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e"}
Nov 02 17:41:27.730856 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.730820    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:27.804633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.804547    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca\") pod \"6dc82128-f84a-4563-9920-af57cf6e2c1c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") "
Nov 02 17:41:27.804802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.804668    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca\") pod \"6dc82128-f84a-4563-9920-af57cf6e2c1c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") "
Nov 02 17:41:27.804802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.804779    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn\" (UniqueName: \"kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn\") pod \"6dc82128-f84a-4563-9920-af57cf6e2c1c\" (UID: \"6dc82128-f84a-4563-9920-af57cf6e2c1c\") "
Nov 02 17:41:27.804950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.804926    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca" pod "6dc82128-f84a-4563-9920-af57cf6e2c1c" (UID: "6dc82128-f84a-4563-9920-af57cf6e2c1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:27.805223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.805198    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:27.807031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.806995    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca" pod "6dc82128-f84a-4563-9920-af57cf6e2c1c" (UID: "6dc82128-f84a-4563-9920-af57cf6e2c1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:27.807151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.807122    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn" pod "6dc82128-f84a-4563-9920-af57cf6e2c1c" (UID: "6dc82128-f84a-4563-9920-af57cf6e2c1c"). InnerVolumeSpecName "kube-api-access-vnnrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:27.906020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.905986    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc82128-f84a-4563-9920-af57cf6e2c1c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:27.906020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:27.906017    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnrn\" (UniqueName: \"kubernetes.io/projected/6dc82128-f84a-4563-9920-af57cf6e2c1c-kube-api-access-vnnrn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:28.056905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.056792    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:28.057063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.057037    5042 patch_prober.go:29] interesting pod/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:41:28.057105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.057083    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 17:41:28.531805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.531769    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"
Nov 02 17:41:28.532249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.531805    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c" event={"ID":"6dc82128-f84a-4563-9920-af57cf6e2c1c","Type":"ContainerDied","Data":"2b9f2f6d84d8af24e3983fb3c3c435ae7aaff9b36b6b46d0e9e49256a0c65a11"}
Nov 02 17:41:28.532249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.531876    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9f2f6d84d8af24e3983fb3c3c435ae7aaff9b36b6b46d0e9e49256a0c65a11"
Nov 02 17:41:28.971164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.971130    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"]
Nov 02 17:41:28.991248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:28.991210    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-38f4eea3-gqdhj"]
Nov 02 17:41:29.056084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:29.056047    5042 patch_prober.go:29] interesting pod/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:41:29.056284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:29.056117    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 17:41:29.132889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:29.132181    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"]
Nov 02 17:41:29.132889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:29.132860    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy" containerID="cri-o://1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954" gracePeriod=30
Nov 02 17:41:29.542966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:29.542936    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="065964b7-8560-4d1d-9e8a-32b5d72f946c" path="/var/lib/kubelet/pods/065964b7-8560-4d1d-9e8a-32b5d72f946c/volumes"
Nov 02 17:41:29.632648 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:29.632609    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"]
Nov 02 17:41:30.056818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.056779    5042 patch_prober.go:29] interesting pod/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:41:30.057020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.056877    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 17:41:30.350877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.350296    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"]
Nov 02 17:41:30.351044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351032    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7d52df2a-e352-4df2-9869-1979bef7dc17" containerName="revoke-keys"
Nov 02 17:41:30.351106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351051    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d52df2a-e352-4df2-9869-1979bef7dc17" containerName="revoke-keys"
Nov 02 17:41:30.351106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351095    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6dc82128-f84a-4563-9920-af57cf6e2c1c" containerName="revoke-keys"
Nov 02 17:41:30.351106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351103    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc82128-f84a-4563-9920-af57cf6e2c1c" containerName="revoke-keys"
Nov 02 17:41:30.351253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351132    5042 cpu_manager.go:414] "containerMap: removing container" podUID="065964b7-8560-4d1d-9e8a-32b5d72f946c" containerName="revoke-keys"
Nov 02 17:41:30.351253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351141    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="065964b7-8560-4d1d-9e8a-32b5d72f946c" containerName="revoke-keys"
Nov 02 17:41:30.351253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351156    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" containerName="revoke-keys"
Nov 02 17:41:30.351253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351163    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" containerName="revoke-keys"
Nov 02 17:41:30.351253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351252    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" containerName="revoke-keys"
Nov 02 17:41:30.351429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351265    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d52df2a-e352-4df2-9869-1979bef7dc17" containerName="revoke-keys"
Nov 02 17:41:30.351429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351279    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc82128-f84a-4563-9920-af57cf6e2c1c" containerName="revoke-keys"
Nov 02 17:41:30.351429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.351291    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="065964b7-8560-4d1d-9e8a-32b5d72f946c" containerName="revoke-keys"
Nov 02 17:41:30.356190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.356166    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.360541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.360500    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-a37cf0-openshift-default-dockercfg-mbxtp\""
Nov 02 17:41:30.407082 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.407042    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"]
Nov 02 17:41:30.440223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440181    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440239    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440378    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440415    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440443    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8nr\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440586    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440632    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440730    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.440824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.440778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.541706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.541660    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.541925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.541739    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.541925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.541777    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.541925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.541826    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.541925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.541874    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.542148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.541963    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.542148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.542006    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.542148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.542037    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8nr\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.542148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.542124    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.543205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.543167    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.544259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.543707    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.544259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.544061    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.545531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.545446    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.546044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.546015    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.547233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.547204    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.547475 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.547452    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.552382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.552332    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.553311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.553281    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8nr\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:30.670084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:30.669996    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:31.001177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.001143    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"]
Nov 02 17:41:31.003522 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:31.003487    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1146d135_49fd_4c59_bf15_ea822c104e1a.slice/crio-60afe0e729819f3824d09c20e942e42fcd7e1f8e92a0c9677034f642086ec718 WatchSource:0}: Error finding container 60afe0e729819f3824d09c20e942e42fcd7e1f8e92a0c9677034f642086ec718: Status 404 returned error can't find the container with id 60afe0e729819f3824d09c20e942e42fcd7e1f8e92a0c9677034f642086ec718
Nov 02 17:41:31.006307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.006266    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:31.006436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.006342    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:31.006436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.006378    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:31.056319 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.056281    5042 patch_prober.go:29] interesting pod/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:41:31.056496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.056347    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 17:41:31.555447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.555408    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" event={"ID":"1146d135-49fd-4c59-bf15-ea822c104e1a","Type":"ContainerStarted","Data":"05cadd8ec1d38d2b6b1f5335d0474fa22a4516f0d06510ef7841cb73393706c3"}
Nov 02 17:41:31.555900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.555475    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" event={"ID":"1146d135-49fd-4c59-bf15-ea822c104e1a","Type":"ContainerStarted","Data":"60afe0e729819f3824d09c20e942e42fcd7e1f8e92a0c9677034f642086ec718"}
Nov 02 17:41:31.598867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.590497    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podStartSLOduration=1.59047636 podStartE2EDuration="1.59047636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:31.58814091 +0000 UTC m=+504.167550297" watchObservedRunningTime="2026-11-02 17:41:31.59047636 +0000 UTC m=+504.169885747"
Nov 02 17:41:31.670706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:31.670670    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:32.056664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:32.056625    5042 patch_prober.go:29] interesting pod/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:41:32.056887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:32.056685    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 17:41:32.684206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:32.684167    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:33.060347 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.060318    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:33.060812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.060794    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:33.061642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.061620    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:33.123150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.123115    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"]
Nov 02 17:41:33.422206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.422119    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"]
Nov 02 17:41:33.424425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.424393    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ae9c41e-zj6cn"]
Nov 02 17:41:33.543171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.543137    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d52df2a-e352-4df2-9869-1979bef7dc17" path="/var/lib/kubelet/pods/7d52df2a-e352-4df2-9869-1979bef7dc17/volumes"
Nov 02 17:41:33.567409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.567382    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:33.568541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.568519    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:33.671439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.671362    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"]
Nov 02 17:41:33.673679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:33.673614    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-868a2ee9-tpbhg"]
Nov 02 17:41:34.415760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.415690    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:34.488589 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488548    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.488799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488604    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.488799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488633    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.488799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488709    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.488799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488740    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.488799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488777    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.489085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488831    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.489085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488899    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.489085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488919    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:34.489085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488969    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:34.489085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.488990    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy\") pod \"75d74498-bf13-4db1-be63-6ba16d9c93dd\" (UID: \"75d74498-bf13-4db1-be63-6ba16d9c93dd\") "
Nov 02 17:41:34.489337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.489191    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:34.489473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.489451    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.489529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.489478    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.489529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.489493    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.489621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.489531    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:34.489621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.489561    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:34.492130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.492101    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:34.492258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.492239    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "kube-api-access-bc8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:34.492338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.492316    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:41:34.492947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.492928    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token" pod "75d74498-bf13-4db1-be63-6ba16d9c93dd" (UID: "75d74498-bf13-4db1-be63-6ba16d9c93dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:34.575432 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.575391    5042 generic.go:356] "Generic (PLEG): container finished" podID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerID="1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954" exitCode=0
Nov 02 17:41:34.575607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.575445    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"
Nov 02 17:41:34.575607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.575468    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" event={"ID":"75d74498-bf13-4db1-be63-6ba16d9c93dd","Type":"ContainerDied","Data":"1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954"}
Nov 02 17:41:34.575607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.575523    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng" event={"ID":"75d74498-bf13-4db1-be63-6ba16d9c93dd","Type":"ContainerDied","Data":"4a6bcf14a32571f7df8469a55e2991c776e21f8d3bbccc48d97b461be0c8c04e"}
Nov 02 17:41:34.575607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.575548    5042 scope.go:122] "RemoveContainer" containerID="1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954"
Nov 02 17:41:34.586767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.586743    5042 scope.go:122] "RemoveContainer" containerID="d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"
Nov 02 17:41:34.590722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.590696    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.590815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.590728    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d74498-bf13-4db1-be63-6ba16d9c93dd-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.590815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.590744    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.590815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.590758    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d74498-bf13-4db1-be63-6ba16d9c93dd-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.590815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.590771    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d74498-bf13-4db1-be63-6ba16d9c93dd-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.590815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.590785    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8kg\" (UniqueName: \"kubernetes.io/projected/75d74498-bf13-4db1-be63-6ba16d9c93dd-kube-api-access-bc8kg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:34.596570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.596547    5042 scope.go:122] "RemoveContainer" containerID="1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954"
Nov 02 17:41:34.596916 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:34.596888    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954\": container with ID starting with 1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954 not found: ID does not exist" containerID="1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954"
Nov 02 17:41:34.596977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.596932    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954"} err="failed to get container status \"1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954\": rpc error: code = NotFound desc = could not find container \"1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954\": container with ID starting with 1db7585262804eec6f76fe63b433f59fb7d4b3ecbb77d5e6bd04ae499ce52954 not found: ID does not exist"
Nov 02 17:41:34.596977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.596957    5042 scope.go:122] "RemoveContainer" containerID="d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"
Nov 02 17:41:34.597237 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:34.597213    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a\": container with ID starting with d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a not found: ID does not exist" containerID="d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"
Nov 02 17:41:34.597277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.597245    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a"} err="failed to get container status \"d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a\": rpc error: code = NotFound desc = could not find container \"d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a\": container with ID starting with d59418d3c9851811f72979fff418c31f6162dc9cb21602d51a780fc23975209a not found: ID does not exist"
Nov 02 17:41:34.604070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.604028    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"]
Nov 02 17:41:34.606594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.606571    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-38f4eea3-gw-openshift-default-7b4dc7dcc-xpcng"]
Nov 02 17:41:34.633615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.633572    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9" podUID="d8e3c225-09e7-4337-be11-9bd8e0b90be8" containerName="payload-processing" containerID="cri-o://2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec" gracePeriod=25
Nov 02 17:41:34.700729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.700633    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"]
Nov 02 17:41:34.908266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:34.908232    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:35.017071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.016985    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c\" (UniqueName: \"kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c\") pod \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") "
Nov 02 17:41:35.017388 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.017137    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume\") pod \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\" (UID: \"d8e3c225-09e7-4337-be11-9bd8e0b90be8\") "
Nov 02 17:41:35.017660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.017628    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume" pod "d8e3c225-09e7-4337-be11-9bd8e0b90be8" (UID: "d8e3c225-09e7-4337-be11-9bd8e0b90be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:35.020032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.020000    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c" pod "d8e3c225-09e7-4337-be11-9bd8e0b90be8" (UID: "d8e3c225-09e7-4337-be11-9bd8e0b90be8"). InnerVolumeSpecName "kube-api-access-5xg2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:35.128498 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.128463    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e3c225-09e7-4337-be11-9bd8e0b90be8-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:35.128498 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.128498    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xg2c\" (UniqueName: \"kubernetes.io/projected/d8e3c225-09e7-4337-be11-9bd8e0b90be8-kube-api-access-5xg2c\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:35.219205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.219166    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"]
Nov 02 17:41:35.224199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.224167    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-55f9886f-b7f7c"]
Nov 02 17:41:35.384204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.384097    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"]
Nov 02 17:41:35.551611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.551575    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc82128-f84a-4563-9920-af57cf6e2c1c" path="/var/lib/kubelet/pods/6dc82128-f84a-4563-9920-af57cf6e2c1c/volumes"
Nov 02 17:41:35.552313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.552292    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" path="/var/lib/kubelet/pods/75d74498-bf13-4db1-be63-6ba16d9c93dd/volumes"
Nov 02 17:41:35.552802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.552774    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b" path="/var/lib/kubelet/pods/c58f3bae-45b4-4c49-a7e1-e844bd9dfa6b/volumes"
Nov 02 17:41:35.585465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.585430    5042 generic.go:356] "Generic (PLEG): container finished" podID="d8e3c225-09e7-4337-be11-9bd8e0b90be8" containerID="2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec" exitCode=0
Nov 02 17:41:35.586182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.585496    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"
Nov 02 17:41:35.586182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.585514    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9" event={"ID":"d8e3c225-09e7-4337-be11-9bd8e0b90be8","Type":"ContainerDied","Data":"2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec"}
Nov 02 17:41:35.586182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.585588    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9" event={"ID":"d8e3c225-09e7-4337-be11-9bd8e0b90be8","Type":"ContainerDied","Data":"854d70583d832c1924e40837de819ebe12fb43edcbae0bf6c16bdffff315a5ab"}
Nov 02 17:41:35.586182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.585632    5042 scope.go:122] "RemoveContainer" containerID="2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec"
Nov 02 17:41:35.586358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.586238    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy" containerID="cri-o://89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e" gracePeriod=30
Nov 02 17:41:35.599964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.599920    5042 scope.go:122] "RemoveContainer" containerID="2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec"
Nov 02 17:41:35.600370 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:35.600345    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec\": container with ID starting with 2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec not found: ID does not exist" containerID="2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec"
Nov 02 17:41:35.600460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.600376    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec"} err="failed to get container status \"2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec\": rpc error: code = NotFound desc = could not find container \"2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec\": container with ID starting with 2a6ed5e83be2426efa5d3fa0fb827230f213c80b2ac6acb60f41344fa79decec not found: ID does not exist"
Nov 02 17:41:35.606250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.606193    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"]
Nov 02 17:41:35.610200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:35.610171    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ae9c41e-5d6dbfc9b7-s69h9"]
Nov 02 17:41:36.607913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.607871    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"]
Nov 02 17:41:36.608651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608568    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d8e3c225-09e7-4337-be11-9bd8e0b90be8" containerName="payload-processing"
Nov 02 17:41:36.608651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608594    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e3c225-09e7-4337-be11-9bd8e0b90be8" containerName="payload-processing"
Nov 02 17:41:36.608651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608637    5042 cpu_manager.go:414] "containerMap: removing container" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy"
Nov 02 17:41:36.608651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608643    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy"
Nov 02 17:41:36.608651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608655    5042 cpu_manager.go:414] "containerMap: removing container" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy"
Nov 02 17:41:36.608651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608660    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy"
Nov 02 17:41:36.609075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608754    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy"
Nov 02 17:41:36.609075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608764    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d74498-bf13-4db1-be63-6ba16d9c93dd" containerName="istio-proxy"
Nov 02 17:41:36.609075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.608773    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e3c225-09e7-4337-be11-9bd8e0b90be8" containerName="payload-processing"
Nov 02 17:41:36.613927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.613899    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.625067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.625039    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-24cfb6-openshift-default-dockercfg-r2v2f\""
Nov 02 17:41:36.626122 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.626095    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"]
Nov 02 17:41:36.754815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.754768    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.754815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.754826    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.754878    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.755013    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.755042    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.755062    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.755161    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.755220    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bg5\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.755302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.755270    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856328 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856288    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856352    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856369    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856397    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856420    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856439    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856499    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856571    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bg5\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856591    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856759    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856822    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.856923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.856903    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.857469 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.857444    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.857695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.857675    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.859633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.859568    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.860180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.860152    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.870319 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.870288    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.870598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.870581    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bg5\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:36.934651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:36.934616    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:37.261277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.261244    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"]
Nov 02 17:41:37.271167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.268876    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:37.271167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.269017    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:37.271167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.269103    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:37.543507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.543404    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e3c225-09e7-4337-be11-9bd8e0b90be8" path="/var/lib/kubelet/pods/d8e3c225-09e7-4337-be11-9bd8e0b90be8/volumes"
Nov 02 17:41:37.603848 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.603807    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" event={"ID":"279aa536-5551-46b2-81e3-656424fe04ad","Type":"ContainerStarted","Data":"6a0e6016bf110e6f3457927e073b6cc59f80b2adc0c0ae8574057c800f6c34ac"}
Nov 02 17:41:37.604022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.603906    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" event={"ID":"279aa536-5551-46b2-81e3-656424fe04ad","Type":"ContainerStarted","Data":"8b300906601cb0017b51d2d6fc25c4ecdbc750715e3e24681efb9c399e65d77b"}
Nov 02 17:41:37.630118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.630050    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podStartSLOduration=1.630027955 podStartE2EDuration="1.630027955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:37.625093829 +0000 UTC m=+510.204503216" watchObservedRunningTime="2026-11-02 17:41:37.630027955 +0000 UTC m=+510.209437343"
Nov 02 17:41:37.935701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:37.935607    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:38.123771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.123719    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p" podUID="92cbc233-d835-4531-b5a4-cede4eeefeca" containerName="payload-pre-processing" containerID="cri-o://9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7" gracePeriod=25
Nov 02 17:41:38.441634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.441606    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:38.573268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.573166    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume\") pod \"92cbc233-d835-4531-b5a4-cede4eeefeca\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") "
Nov 02 17:41:38.573268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.573217    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z\" (UniqueName: \"kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z\") pod \"92cbc233-d835-4531-b5a4-cede4eeefeca\" (UID: \"92cbc233-d835-4531-b5a4-cede4eeefeca\") "
Nov 02 17:41:38.573641 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.573615    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume" pod "92cbc233-d835-4531-b5a4-cede4eeefeca" (UID: "92cbc233-d835-4531-b5a4-cede4eeefeca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:38.573764 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.573749    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cbc233-d835-4531-b5a4-cede4eeefeca-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:38.575569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.575540    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z" pod "92cbc233-d835-4531-b5a4-cede4eeefeca" (UID: "92cbc233-d835-4531-b5a4-cede4eeefeca"). InnerVolumeSpecName "kube-api-access-8nx5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:38.611068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.611035    5042 generic.go:356] "Generic (PLEG): container finished" podID="92cbc233-d835-4531-b5a4-cede4eeefeca" containerID="9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7" exitCode=0
Nov 02 17:41:38.611247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.611083    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"
Nov 02 17:41:38.611247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.611130    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p" event={"ID":"92cbc233-d835-4531-b5a4-cede4eeefeca","Type":"ContainerDied","Data":"9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7"}
Nov 02 17:41:38.611247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.611187    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p" event={"ID":"92cbc233-d835-4531-b5a4-cede4eeefeca","Type":"ContainerDied","Data":"fc7c8c4c80ea7104b56c96edcae77e9a96da13f817ca3b687beb7d528d9ee371"}
Nov 02 17:41:38.611247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.611217    5042 scope.go:122] "RemoveContainer" containerID="9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7"
Nov 02 17:41:38.621444 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.621422    5042 scope.go:122] "RemoveContainer" containerID="9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7"
Nov 02 17:41:38.621818 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:38.621796    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7\": container with ID starting with 9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7 not found: ID does not exist" containerID="9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7"
Nov 02 17:41:38.621885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.621828    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7"} err="failed to get container status \"9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7\": rpc error: code = NotFound desc = could not find container \"9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7\": container with ID starting with 9bc8960177890925a027d268d2bcf56f467853ae2e48b68b3bb974faee145cd7 not found: ID does not exist"
Nov 02 17:41:38.634901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.634866    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"]
Nov 02 17:41:38.639246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.639211    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad831efe-647887bb4c-bk82p"]
Nov 02 17:41:38.675971 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.675938    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nx5z\" (UniqueName: \"kubernetes.io/projected/92cbc233-d835-4531-b5a4-cede4eeefeca-kube-api-access-8nx5z\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:38.940937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:38.940906    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:39.546741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:39.546712    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92cbc233-d835-4531-b5a4-cede4eeefeca" path="/var/lib/kubelet/pods/92cbc233-d835-4531-b5a4-cede4eeefeca/volumes"
Nov 02 17:41:39.619511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:39.619475    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:39.620724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:39.620699    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:41:39.701776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:39.701701    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k" podUID="61396460-12ef-4e0a-a5c4-2aca8b73f37c" containerName="payload-pre-processing" containerID="cri-o://158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a" gracePeriod=25
Nov 02 17:41:40.028286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.028249    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:40.099650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.099609    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume\") pod \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") "
Nov 02 17:41:40.099887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.099787    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb\" (UniqueName: \"kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb\") pod \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\" (UID: \"61396460-12ef-4e0a-a5c4-2aca8b73f37c\") "
Nov 02 17:41:40.100083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.100049    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume" pod "61396460-12ef-4e0a-a5c4-2aca8b73f37c" (UID: "61396460-12ef-4e0a-a5c4-2aca8b73f37c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:40.100326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.100298    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61396460-12ef-4e0a-a5c4-2aca8b73f37c-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:40.102219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.102193    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb" pod "61396460-12ef-4e0a-a5c4-2aca8b73f37c" (UID: "61396460-12ef-4e0a-a5c4-2aca8b73f37c"). InnerVolumeSpecName "kube-api-access-5qzfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:40.202943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.201582    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzfb\" (UniqueName: \"kubernetes.io/projected/61396460-12ef-4e0a-a5c4-2aca8b73f37c-kube-api-access-5qzfb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:40.628015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.627981    5042 generic.go:356] "Generic (PLEG): container finished" podID="61396460-12ef-4e0a-a5c4-2aca8b73f37c" containerID="158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a" exitCode=0
Nov 02 17:41:40.628231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.628054    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"
Nov 02 17:41:40.628231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.628071    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k" event={"ID":"61396460-12ef-4e0a-a5c4-2aca8b73f37c","Type":"ContainerDied","Data":"158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a"}
Nov 02 17:41:40.628231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.628143    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k" event={"ID":"61396460-12ef-4e0a-a5c4-2aca8b73f37c","Type":"ContainerDied","Data":"7021f21f24f892f68f49b64593305a308429a8939dfd618c54ac305ab84cb890"}
Nov 02 17:41:40.628231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.628174    5042 scope.go:122] "RemoveContainer" containerID="158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a"
Nov 02 17:41:40.709345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.709310    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"]
Nov 02 17:41:40.711775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.711746    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-55f9886f-76ff44474c-9k48k"]
Nov 02 17:41:40.716087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.715970    5042 scope.go:122] "RemoveContainer" containerID="158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a"
Nov 02 17:41:40.716365 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:40.716340    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a\": container with ID starting with 158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a not found: ID does not exist" containerID="158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a"
Nov 02 17:41:40.716466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.716373    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a"} err="failed to get container status \"158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a\": rpc error: code = NotFound desc = could not find container \"158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a\": container with ID starting with 158679917b09bd38556669d665f08adac24681368321f7e9b9013b560525844a not found: ID does not exist"
Nov 02 17:41:40.945765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:40.945734    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:41.124177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124136    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124190    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124218    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124308    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124346    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124407    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124477    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124531    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124510    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:41.124590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124565    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data\") pod \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\" (UID: \"a4ce0beb-a0e5-4929-98fb-5b39970c6c99\") "
Nov 02 17:41:41.124899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124585    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:41.124899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124678    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:41.124899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124743    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:41.125035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.124936    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:41.125196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.125053    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.125250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.125204    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.125250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.125229    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.125250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.125243    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.126655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.126628    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:41:41.126905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.126884    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:41.127154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.127137    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "kube-api-access-6xsjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:41.127221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.127202    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy" pod "a4ce0beb-a0e5-4929-98fb-5b39970c6c99" (UID: "a4ce0beb-a0e5-4929-98fb-5b39970c6c99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:41:41.226107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.226070    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsjp\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-kube-api-access-6xsjp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.226107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.226107    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.226323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.226123    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.226323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.226136    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.226323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.226149    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4ce0beb-a0e5-4929-98fb-5b39970c6c99-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:41:41.542627 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.542541    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61396460-12ef-4e0a-a5c4-2aca8b73f37c" path="/var/lib/kubelet/pods/61396460-12ef-4e0a-a5c4-2aca8b73f37c/volumes"
Nov 02 17:41:41.637027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.636993    5042 generic.go:356] "Generic (PLEG): container finished" podID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerID="89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e" exitCode=0
Nov 02 17:41:41.637247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.637058    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"
Nov 02 17:41:41.637247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.637083    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" event={"ID":"a4ce0beb-a0e5-4929-98fb-5b39970c6c99","Type":"ContainerDied","Data":"89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e"}
Nov 02 17:41:41.637247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.637149    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6" event={"ID":"a4ce0beb-a0e5-4929-98fb-5b39970c6c99","Type":"ContainerDied","Data":"11b71d382b45b17ac471953125840dff9205227799c85fa8f75538820d7ebdc9"}
Nov 02 17:41:41.637247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.637180    5042 scope.go:122] "RemoveContainer" containerID="89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e"
Nov 02 17:41:41.648656 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.648630    5042 scope.go:122] "RemoveContainer" containerID="4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"
Nov 02 17:41:41.657211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.657181    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"]
Nov 02 17:41:41.660425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.658691    5042 scope.go:122] "RemoveContainer" containerID="89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e"
Nov 02 17:41:41.660425 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:41.659134    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e\": container with ID starting with 89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e not found: ID does not exist" containerID="89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e"
Nov 02 17:41:41.660425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.659187    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e"} err="failed to get container status \"89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e\": rpc error: code = NotFound desc = could not find container \"89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e\": container with ID starting with 89d7e33314c070cd69abee29d61e8ebacc6c69f43f95f103888abc969195578e not found: ID does not exist"
Nov 02 17:41:41.660425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.659208    5042 scope.go:122] "RemoveContainer" containerID="4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"
Nov 02 17:41:41.660425 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:41.659529    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591\": container with ID starting with 4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591 not found: ID does not exist" containerID="4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"
Nov 02 17:41:41.660425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.659550    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591"} err="failed to get container status \"4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591\": rpc error: code = NotFound desc = could not find container \"4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591\": container with ID starting with 4a08c2e626715620c10d342a00b6787c86b17b8a900f3fd0346ea26192e8c591 not found: ID does not exist"
Nov 02 17:41:41.662171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:41.662145    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-55f9886f-openshift-default-78956b55fc-g5dk6"]
Nov 02 17:41:43.551397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:43.551180    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" path="/var/lib/kubelet/pods/a4ce0beb-a0e5-4929-98fb-5b39970c6c99/volumes"
Nov 02 17:41:48.544813 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:41:48.544771    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:41:49.613103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:49.613042    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:41:49.614036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:49.613688    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:41:50.736189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.736151    5042 generic.go:356] "Generic (PLEG): container finished" podID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerID="05cadd8ec1d38d2b6b1f5335d0474fa22a4516f0d06510ef7841cb73393706c3" exitCode=0
Nov 02 17:41:50.736637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.736231    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" event={"ID":"1146d135-49fd-4c59-bf15-ea822c104e1a","Type":"ContainerDied","Data":"05cadd8ec1d38d2b6b1f5335d0474fa22a4516f0d06510ef7841cb73393706c3"}
Nov 02 17:41:50.736757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.736740    5042 scope.go:122] "RemoveContainer" containerID="05cadd8ec1d38d2b6b1f5335d0474fa22a4516f0d06510ef7841cb73393706c3"
Nov 02 17:41:50.738366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.738340    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:50.738458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.738408    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:50.738458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.738440    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:50.944207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944170    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"]
Nov 02 17:41:50.944702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944682    5042 cpu_manager.go:414] "containerMap: removing container" podUID="61396460-12ef-4e0a-a5c4-2aca8b73f37c" containerName="payload-pre-processing"
Nov 02 17:41:50.944702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944702    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61396460-12ef-4e0a-a5c4-2aca8b73f37c" containerName="payload-pre-processing"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944719    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944724    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944734    5042 cpu_manager.go:414] "containerMap: removing container" podUID="92cbc233-d835-4531-b5a4-cede4eeefeca" containerName="payload-pre-processing"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944739    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92cbc233-d835-4531-b5a4-cede4eeefeca" containerName="payload-pre-processing"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944820    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944829    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="61396460-12ef-4e0a-a5c4-2aca8b73f37c" containerName="payload-pre-processing"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944854    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy"
Nov 02 17:41:50.944914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.944862    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="92cbc233-d835-4531-b5a4-cede4eeefeca" containerName="payload-pre-processing"
Nov 02 17:41:50.952901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.952130    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:50.965878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.963978    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"]
Nov 02 17:41:50.965878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.964944    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"default-dockercfg-k8pxt\""
Nov 02 17:41:50.965878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.965037    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"openshift-service-ca.crt\""
Nov 02 17:41:50.965878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.965124    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"kube-root-ca.crt\""
Nov 02 17:41:50.965878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.965257    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-a37cf0\"/\"simulator-w3-a37cf0-kserve-self-signed-certs\""
Nov 02 17:41:50.991932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.971617    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8sw\" (UniqueName: \"kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:50.991932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.971658    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:50.991932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.971686    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:50.991932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.971706    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:50.991932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.971768    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:50.991932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:50.971784    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.073471    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.073528    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.073617    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8sw\" (UniqueName: \"kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.073666    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.073707    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.073731    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.074205    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.126892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.085536    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.126892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.085880    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.126892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.101544    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.126892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.126355    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8sw\" (UniqueName: \"kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.133637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.133272    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm\") pod \"simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") " pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.271802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.271675    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:51.629944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.629909    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"]
Nov 02 17:41:51.754697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.754652    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" event={"ID":"1146d135-49fd-4c59-bf15-ea822c104e1a","Type":"ContainerStarted","Data":"163a1e0a44f44cdf8289769a70ccbac735501053f844f786372be9b61a659a12"}
Nov 02 17:41:51.756416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:51.756374    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" event={"ID":"6a4c8897-3567-48cc-84fd-1f6f6ab64245","Type":"ContainerStarted","Data":"b5aa627a45ac5966d16bc8e745f83f54d38b9d538f0ad66a3cd4c8088dca82e0"}
Nov 02 17:41:52.670660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:52.670627    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:52.670951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:52.670821    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 17:41:52.671034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:52.670978    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 17:41:52.764716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:52.764677    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" event={"ID":"6a4c8897-3567-48cc-84fd-1f6f6ab64245","Type":"ContainerStarted","Data":"a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575"}
Nov 02 17:41:52.820863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:52.817988    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podStartSLOduration=2.817961519 podStartE2EDuration="2.817961519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:52.793284107 +0000 UTC m=+525.372693494" watchObservedRunningTime="2026-11-02 17:41:52.817961519 +0000 UTC m=+525.397370905"
Nov 02 17:41:53.275865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:53.273954    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:41:53.671419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:53.671335    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 17:41:53.671419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:53.671407    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 17:41:54.670638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:54.670541    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 17:41:54.670638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:54.670596    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 17:41:55.664129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:55.664030    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:41:55.664129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:55.664105    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:41:55.671073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:55.671037    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 17:41:55.671616 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:55.671087    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 17:41:56.550353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.550316    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"]
Nov 02 17:41:56.551065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.551044    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy"
Nov 02 17:41:56.551111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.551069    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4ce0beb-a0e5-4929-98fb-5b39970c6c99" containerName="istio-proxy"
Nov 02 17:41:56.557686 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.557648    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.565372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.565338    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"]
Nov 02 17:41:56.567017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.566916    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"default-dockercfg-7nb7p\""
Nov 02 17:41:56.567161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.567016    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"simulator-w2-66eed1-kserve-self-signed-certs\""
Nov 02 17:41:56.623704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.623671    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"openshift-service-ca.crt\""
Nov 02 17:41:56.623704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.623675    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"kube-root-ca.crt\""
Nov 02 17:41:56.671049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.671013    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 02 17:41:56.671229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.671078    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 02 17:41:56.713942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.713903    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.713942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.713947    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.714218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.713993    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.714218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.714142    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.714218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.714176    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.714379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.714307    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxxq\" (UniqueName: \"kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815004    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxxq\" (UniqueName: \"kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815104    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815139    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815170    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815225    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815261    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.815718    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.816363    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.816927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.816562    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.828014 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.827949    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.839996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.839707    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.842738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.842710    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxxq\" (UniqueName: \"kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq\") pod \"simulator-w2-66eed1-kserve-85486cc46b-mthb8\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") " pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:56.899407 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:56.899099    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:57.195061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.195034    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"]
Nov 02 17:41:57.197233 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:41:57.197192    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3973d45d_c19f_4da3_9795_ce61ad4f9fd4.slice/crio-0ebe44f513e7ebb44b20d5b4c54c8837eb859ff91b8e87208ecebcf9e74e4f48 WatchSource:0}: Error finding container 0ebe44f513e7ebb44b20d5b4c54c8837eb859ff91b8e87208ecebcf9e74e4f48: Status 404 returned error can't find the container with id 0ebe44f513e7ebb44b20d5b4c54c8837eb859ff91b8e87208ecebcf9e74e4f48
Nov 02 17:41:57.839100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.839060    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" event={"ID":"3973d45d-c19f-4da3-9795-ce61ad4f9fd4","Type":"ContainerStarted","Data":"08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef"}
Nov 02 17:41:57.839585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.839119    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" event={"ID":"3973d45d-c19f-4da3-9795-ce61ad4f9fd4","Type":"ContainerStarted","Data":"0ebe44f513e7ebb44b20d5b4c54c8837eb859ff91b8e87208ecebcf9e74e4f48"}
Nov 02 17:41:57.859956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.859897    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podStartSLOduration=1.8598789949999999 podStartE2EDuration="1.859878995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:57.856658342 +0000 UTC m=+530.436067728" watchObservedRunningTime="2026-11-02 17:41:57.859878995 +0000 UTC m=+530.439288383"
Nov 02 17:41:57.866538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.866504    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:57.867166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.867146    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:57.868639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.868615    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 17:41:57.900670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:57.900640    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:41:59.854977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:59.854939    5042 generic.go:356] "Generic (PLEG): container finished" podID="279aa536-5551-46b2-81e3-656424fe04ad" containerID="6a0e6016bf110e6f3457927e073b6cc59f80b2adc0c0ae8574057c800f6c34ac" exitCode=0
Nov 02 17:41:59.855499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:59.855019    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" event={"ID":"279aa536-5551-46b2-81e3-656424fe04ad","Type":"ContainerDied","Data":"6a0e6016bf110e6f3457927e073b6cc59f80b2adc0c0ae8574057c800f6c34ac"}
Nov 02 17:41:59.856086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:59.856060    5042 scope.go:122] "RemoveContainer" containerID="6a0e6016bf110e6f3457927e073b6cc59f80b2adc0c0ae8574057c800f6c34ac"
Nov 02 17:41:59.863682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:59.863638    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:59.863965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:59.863735    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:59.864085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:41:59.864059    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:00.870281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:00.870241    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" event={"ID":"279aa536-5551-46b2-81e3-656424fe04ad","Type":"ContainerStarted","Data":"e2933dbb106f482dbc860c61ede7068bb975999a22d62a09ac36940d7b77ef08"}
Nov 02 17:42:00.935494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:00.935455    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:42:00.935732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:00.935707    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 17:42:00.935787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:00.935764    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 17:42:01.282236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:01.282189    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:42:01.318638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:01.318605    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:42:01.920166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:01.920136    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:42:01.935382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:01.935339    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 17:42:01.935580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:01.935401    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 17:42:02.935236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:02.935197    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 17:42:02.935674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:02.935264    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 17:42:03.935606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:03.935567    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 17:42:03.936061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:03.935628    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 17:42:04.935575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:04.935542    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 17:42:04.935767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:04.935597    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 17:42:05.935866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:05.935804    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Nov 02 17:42:05.936345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:05.935885    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Nov 02 17:42:06.900052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:06.900017    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:42:06.912936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:06.912910    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:42:06.939372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:06.939342    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:42:06.939892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:06.939873    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:42:06.940645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:06.940621    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 17:42:06.945816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:06.945797    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:42:09.261183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.258097    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"]
Nov 02 17:42:09.268540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.268510    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"]
Nov 02 17:42:09.268932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.268816    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.289762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.289327    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"kube-root-ca.crt\""
Nov 02 17:42:09.290187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.290166    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"default-dockercfg-8h6zz\""
Nov 02 17:42:09.290429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.290399    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"openshift-service-ca.crt\""
Nov 02 17:42:09.290573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.290554    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"simulator-w4-77f103-kserve-self-signed-certs\""
Nov 02 17:42:09.399579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.399536    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.399788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.399636    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzh6\" (UniqueName: \"kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.399788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.399711    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.399788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.399746    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.399788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.399776    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.400094 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.399804    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.501356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.501319    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzh6\" (UniqueName: \"kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.501556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.501382    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.501556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.501413    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.501556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.501430    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.501556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.501446    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.501783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.501563    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.502035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.502004    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.502125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.502108    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.502759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.502734    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.504123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.504096    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.504301 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.504285    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.510313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.510292    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzh6\" (UniqueName: \"kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6\") pod \"simulator-w4-77f103-kserve-f5f7dffc7-wbz7x\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") " pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.598162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.598075    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:09.877007 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.876930    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"]
Nov 02 17:42:09.881309 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:42:09.881273    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod329a6854_eaf7_4e3f_b080_ad6239cde483.slice/crio-9fa687fadfd6b431604ee75561ea84ed3e0d05a2bce0da5cfc1e267c7f85f59e WatchSource:0}: Error finding container 9fa687fadfd6b431604ee75561ea84ed3e0d05a2bce0da5cfc1e267c7f85f59e: Status 404 returned error can't find the container with id 9fa687fadfd6b431604ee75561ea84ed3e0d05a2bce0da5cfc1e267c7f85f59e
Nov 02 17:42:09.954754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:09.954729    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" event={"ID":"329a6854-eaf7-4e3f-b080-ad6239cde483","Type":"ContainerStarted","Data":"9fa687fadfd6b431604ee75561ea84ed3e0d05a2bce0da5cfc1e267c7f85f59e"}
Nov 02 17:42:10.963368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:10.963330    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" event={"ID":"329a6854-eaf7-4e3f-b080-ad6239cde483","Type":"ContainerStarted","Data":"c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3"}
Nov 02 17:42:10.984331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:10.984282    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podStartSLOduration=1.984267505 podStartE2EDuration="1.984267505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:10.982692527 +0000 UTC m=+543.562101941" watchObservedRunningTime="2026-11-02 17:42:10.984267505 +0000 UTC m=+543.563676891"
Nov 02 17:42:11.599092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:11.599049    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:18.553865 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:42:18.552999    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:42:19.598404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:19.598363    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:19.611227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:19.611197    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:20.028078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:20.028049    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:42:27.466216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.466176    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"]
Nov 02 17:42:27.471559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.471531    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.481960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.481930    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"kube-root-ca.crt\""
Nov 02 17:42:27.484240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.484210    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"default-dockercfg-gdgnd\""
Nov 02 17:42:27.484373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.484330    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"openshift-service-ca.crt\""
Nov 02 17:42:27.484976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.484945    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-24cfb6\"/\"premium-w1-24cfb6-kserve-self-signed-certs\""
Nov 02 17:42:27.486502 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.486449    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"]
Nov 02 17:42:27.621584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.621538    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.622071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.622018    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.622220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.622083    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.622220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.622109    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm692\" (UniqueName: \"kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.622220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.622157    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.622220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.622179    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.723908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.723689    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.723908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.723830    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.723908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.723904    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.724211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.723928    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm692\" (UniqueName: \"kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.724211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.724018    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.724211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.724041    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.725461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.725434    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.725623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.725462    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.726080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.726015    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.727982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.727958    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.728079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.728003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.738678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.738637    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm692\" (UniqueName: \"kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692\") pod \"premium-w1-24cfb6-kserve-644cbd56d4-rfvf6\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") " pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:27.792587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:27.792268    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:28.092704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:28.092671    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"]
Nov 02 17:42:28.096979 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:42:28.096946    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72525843_3f51_443b_81d5_b146ebd118fb.slice/crio-2961f3aa55fa15192a4f692db62681799cef9777debd6e56b854737e44756f31 WatchSource:0}: Error finding container 2961f3aa55fa15192a4f692db62681799cef9777debd6e56b854737e44756f31: Status 404 returned error can't find the container with id 2961f3aa55fa15192a4f692db62681799cef9777debd6e56b854737e44756f31
Nov 02 17:42:29.098663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:29.098365    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" event={"ID":"72525843-3f51-443b-81d5-b146ebd118fb","Type":"ContainerStarted","Data":"a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148"}
Nov 02 17:42:29.098663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:29.098415    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" event={"ID":"72525843-3f51-443b-81d5-b146ebd118fb","Type":"ContainerStarted","Data":"2961f3aa55fa15192a4f692db62681799cef9777debd6e56b854737e44756f31"}
Nov 02 17:42:29.120399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:29.120325    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podStartSLOduration=2.120305819 podStartE2EDuration="2.120305819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:29.117080067 +0000 UTC m=+561.696489453" watchObservedRunningTime="2026-11-02 17:42:29.120305819 +0000 UTC m=+561.699715205"
Nov 02 17:42:29.793449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:29.793406    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:30.878363 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.878327    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"]
Nov 02 17:42:30.883933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.883905    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:30.895573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.895540    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"]
Nov 02 17:42:30.896379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.896354    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-77f103\"/\"premium-w4-77f103-kserve-self-signed-certs\""
Nov 02 17:42:30.960080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.960041    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:30.960268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.960097    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:30.960268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.960193    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:30.960268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.960248    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:30.960425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.960307    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhnd\" (UniqueName: \"kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:30.960493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:30.960468    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.061556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061512    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.061766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061594    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.061766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061678    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.061766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061706    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhnd\" (UniqueName: \"kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.061962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061797    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.061962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061854    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.062076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.061958    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.062154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.062130    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.062217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.062200    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.064165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.064138    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.064607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.064591    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.070672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.070652    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhnd\" (UniqueName: \"kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd\") pod \"premium-w4-77f103-kserve-5ccfbc56fb-vtxvw\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") " pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.206624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.206585    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:31.494016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:31.493979    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"]
Nov 02 17:42:31.494178 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:42:31.494153    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod831a619e_ccd4_40b5_9903_e83f3daca0c8.slice/crio-abd8824d21b4ba508016b092449e037908dd2ed6ea52f22c7f911d0dc7a1be0d WatchSource:0}: Error finding container abd8824d21b4ba508016b092449e037908dd2ed6ea52f22c7f911d0dc7a1be0d: Status 404 returned error can't find the container with id abd8824d21b4ba508016b092449e037908dd2ed6ea52f22c7f911d0dc7a1be0d
Nov 02 17:42:32.122190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:32.122149    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" event={"ID":"831a619e-ccd4-40b5-9903-e83f3daca0c8","Type":"ContainerStarted","Data":"b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90"}
Nov 02 17:42:32.122640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:32.122205    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" event={"ID":"831a619e-ccd4-40b5-9903-e83f3daca0c8","Type":"ContainerStarted","Data":"abd8824d21b4ba508016b092449e037908dd2ed6ea52f22c7f911d0dc7a1be0d"}
Nov 02 17:42:32.140955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:32.140905    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podStartSLOduration=2.140891956 podStartE2EDuration="2.140891956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:32.140653124 +0000 UTC m=+564.720062511" watchObservedRunningTime="2026-11-02 17:42:32.140891956 +0000 UTC m=+564.720301341"
Nov 02 17:42:32.207391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:32.207345    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:35.220716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.220674    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"]
Nov 02 17:42:35.226599 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.226573    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.236183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.236152    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-66eed1-openshift-default-dockercfg-dd2rz\""
Nov 02 17:42:35.242376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.242346    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"]
Nov 02 17:42:35.318553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318505    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.318761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318603    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.318761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318634    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.318761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318684    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.318761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318721    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.318761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318753    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.319077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318783    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.319077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318885    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.319077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.318950    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.425390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.425342    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.426721 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.426668    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.426904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.426883    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.427036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.427021    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.427555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.427535    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.428020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.428005    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.428556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.428539    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.429371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.429354    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.429819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.429802    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.430404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.429686    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.430542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.429298    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.430676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.430292    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.430791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.427949    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.430970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.430953    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.433433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.433410    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.451478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.451435    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.454681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.454647    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.455296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.455270    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.555892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.555782    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:35.991863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:35.991609    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"]
Nov 02 17:42:36.047265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:36.047206    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:36.047425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:36.047364    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:36.047491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:36.047453    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:36.166590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:36.166547    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" event={"ID":"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9","Type":"ContainerStarted","Data":"8946a90b4d46949f4f8799fd275a802ea7095c1aafc738b75cfd9742fe910aae"}
Nov 02 17:42:37.178406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.178365    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" event={"ID":"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9","Type":"ContainerStarted","Data":"1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7"}
Nov 02 17:42:37.206460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.205909    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podStartSLOduration=2.20588975 podStartE2EDuration="2.20588975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:37.204290948 +0000 UTC m=+569.783700329" watchObservedRunningTime="2026-11-02 17:42:37.20588975 +0000 UTC m=+569.785299135"
Nov 02 17:42:37.559053 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.559022    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:37.560673 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.560651    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:37.560763 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.560700    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 17:42:37.793874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.793270    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:37.812751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:37.812659    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:38.212796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:38.212764    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:42:38.560191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:38.560097    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:38.560191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:38.560176    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 17:42:39.559040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:39.559004    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:39.559458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:39.559068    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 17:42:40.559762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:40.559722    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:40.560255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:40.559791    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 17:42:41.207248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:41.207214    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:41.224825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:41.224793    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:41.237940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:41.237911    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:42:41.559044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:41.558958    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:41.559044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:41.559013    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 17:42:42.559205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:42.559167    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:42.559714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:42.559228    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 17:42:43.564630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:43.564599    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:43.565123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:43.565095    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:43.565873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:43.565833    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 17:42:48.538391 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:42:48.538303    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:42:53.641734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.641699    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"]
Nov 02 17:42:53.645674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.645647    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.656186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.656160    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:42:53.658182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.658158    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:42:53.658417 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.658393    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:42:53.658522 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.658453    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:42:53.660887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.660865    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"]
Nov 02 17:42:53.705392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.705359    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.705568 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.705527    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.705631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.705564    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdqp\" (UniqueName: \"kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.807020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.806982    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.807020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.807037    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdqp\" (UniqueName: \"kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.807279 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.807107    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.807770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.807743    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.809757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.809738    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.823060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.823021    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdqp\" (UniqueName: \"kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp\") pod \"maas-api-revoke-sub-986864f0-x2grq\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:53.963272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:53.963193    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:54.288295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:54.288264    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"]
Nov 02 17:42:54.329018 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:54.328972    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq" event={"ID":"352b62f2-aa75-44d4-a7a5-62f3067a5bb8","Type":"ContainerStarted","Data":"f8d4d51ca2135b7bcf02b63f1909514c8d403ff48c19ecf36a61b1de711f888b"}
Nov 02 17:42:55.338414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:55.338375    5042 generic.go:356] "Generic (PLEG): container finished" podID="352b62f2-aa75-44d4-a7a5-62f3067a5bb8" containerID="b54ae4dc3b0a421d017224906a677b15481f6924d31930bc24263603401d5d0c" exitCode=0
Nov 02 17:42:55.338962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:55.338480    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq" event={"ID":"352b62f2-aa75-44d4-a7a5-62f3067a5bb8","Type":"ContainerDied","Data":"b54ae4dc3b0a421d017224906a677b15481f6924d31930bc24263603401d5d0c"}
Nov 02 17:42:56.503766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.503733    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:56.556713 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.556673    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca\") pod \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") "
Nov 02 17:42:56.556944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.556724    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca\") pod \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") "
Nov 02 17:42:56.557035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.556953    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp\" (UniqueName: \"kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp\") pod \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\" (UID: \"352b62f2-aa75-44d4-a7a5-62f3067a5bb8\") "
Nov 02 17:42:56.557166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.557141    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca" pod "352b62f2-aa75-44d4-a7a5-62f3067a5bb8" (UID: "352b62f2-aa75-44d4-a7a5-62f3067a5bb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:42:56.557413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.557374    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:42:56.559734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.559708    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca" pod "352b62f2-aa75-44d4-a7a5-62f3067a5bb8" (UID: "352b62f2-aa75-44d4-a7a5-62f3067a5bb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:42:56.559960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.559930    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp" pod "352b62f2-aa75-44d4-a7a5-62f3067a5bb8" (UID: "352b62f2-aa75-44d4-a7a5-62f3067a5bb8"). InnerVolumeSpecName "kube-api-access-cpdqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:42:56.658409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.658327    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpdqp\" (UniqueName: \"kubernetes.io/projected/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-kube-api-access-cpdqp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:42:56.658409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:56.658359    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/352b62f2-aa75-44d4-a7a5-62f3067a5bb8-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:42:57.353955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:57.353921    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"
Nov 02 17:42:57.354141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:57.353949    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq" event={"ID":"352b62f2-aa75-44d4-a7a5-62f3067a5bb8","Type":"ContainerDied","Data":"f8d4d51ca2135b7bcf02b63f1909514c8d403ff48c19ecf36a61b1de711f888b"}
Nov 02 17:42:57.354141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:42:57.354013    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d4d51ca2135b7bcf02b63f1909514c8d403ff48c19ecf36a61b1de711f888b"
Nov 02 17:43:04.048420 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.048382    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"]
Nov 02 17:43:04.048953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.048942    5042 cpu_manager.go:414] "containerMap: removing container" podUID="352b62f2-aa75-44d4-a7a5-62f3067a5bb8" containerName="revoke-keys"
Nov 02 17:43:04.049000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.048956    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352b62f2-aa75-44d4-a7a5-62f3067a5bb8" containerName="revoke-keys"
Nov 02 17:43:04.049085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.049074    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="352b62f2-aa75-44d4-a7a5-62f3067a5bb8" containerName="revoke-keys"
Nov 02 17:43:04.052580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.052565    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.057666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.057639    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:43:04.057811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.057743    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:43:04.057811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.057777    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:43:04.060335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.059683    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:04.063198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.063173    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"]
Nov 02 17:43:04.159029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.158987    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.159287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.159265    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.159370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.159313    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkn5\" (UniqueName: \"kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.261803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.261598    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.261803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.261688    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkn5\" (UniqueName: \"kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.261803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.261751    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.263631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.263204    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.271943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.271911    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.273369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.273327    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkn5\" (UniqueName: \"kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5\") pod \"maas-api-revoke-sub-715178e5-jqq78\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.369413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.369335    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:04.540723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.540686    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"]
Nov 02 17:43:04.545720 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.545695    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.555428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.554609    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"]
Nov 02 17:43:04.659173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.659147    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"]
Nov 02 17:43:04.660566 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:04.660537    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7df942_dcc0_4a93_ba77_2b1b8e86995a.slice/crio-4a7c254de0e481132759824eca7cc570e17be2d8527f9a65a5009e317e16b5c1 WatchSource:0}: Error finding container 4a7c254de0e481132759824eca7cc570e17be2d8527f9a65a5009e317e16b5c1: Status 404 returned error can't find the container with id 4a7c254de0e481132759824eca7cc570e17be2d8527f9a65a5009e317e16b5c1
Nov 02 17:43:04.667136 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.667100    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwmkx\" (UniqueName: \"kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.667248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.667161    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.667422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.667393    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.768765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.768733    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.768965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.768777    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwmkx\" (UniqueName: \"kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.768965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.768830    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.769534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.769512    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.771368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.771349    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.777697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.777673    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwmkx\" (UniqueName: \"kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx\") pod \"maas-api-revoke-sub-ead5a273-p28bf\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:04.864792 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:04.862801    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:05.149311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.149284    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"]
Nov 02 17:43:05.150491 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:05.150467    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636f30cc_655b_4e47_9d3b_c48c980db943.slice/crio-826da41368ac2219938d00e3bb9f90223e85902721e1cababfd5df67dff0b077 WatchSource:0}: Error finding container 826da41368ac2219938d00e3bb9f90223e85902721e1cababfd5df67dff0b077: Status 404 returned error can't find the container with id 826da41368ac2219938d00e3bb9f90223e85902721e1cababfd5df67dff0b077
Nov 02 17:43:05.412653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.412611    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78" event={"ID":"ab7df942-dcc0-4a93-ba77-2b1b8e86995a","Type":"ContainerStarted","Data":"ce7c0debac81e357f9aa26d2cd501773f1f36d4524024cfa4b1590866124d20a"}
Nov 02 17:43:05.412831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.412672    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78" event={"ID":"ab7df942-dcc0-4a93-ba77-2b1b8e86995a","Type":"ContainerStarted","Data":"4a7c254de0e481132759824eca7cc570e17be2d8527f9a65a5009e317e16b5c1"}
Nov 02 17:43:05.414086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.414060    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf" event={"ID":"636f30cc-655b-4e47-9d3b-c48c980db943","Type":"ContainerStarted","Data":"826da41368ac2219938d00e3bb9f90223e85902721e1cababfd5df67dff0b077"}
Nov 02 17:43:05.435809 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.435707    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78" podStartSLOduration=1.435693824 podStartE2EDuration="1.435693824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:05.433471067 +0000 UTC m=+598.012880454" watchObservedRunningTime="2026-11-02 17:43:05.435693824 +0000 UTC m=+598.015103209"
Nov 02 17:43:05.546594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.546557    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"]
Nov 02 17:43:05.550580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.550559    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.559488 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.559463    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b97c6-openshift-default-dockercfg-smxvn\""
Nov 02 17:43:05.562397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.562371    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"]
Nov 02 17:43:05.681945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.681903    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.681958    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.681982    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.682128    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.682193    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682347 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.682289    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682347 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.682330    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.682494    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgcb8\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.682647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.682625    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783604    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgcb8\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783677    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783711    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783754    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783781    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783820    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.783898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783883    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.784224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783921    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.784224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.783949    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.784224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.784176    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.784387 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.784358    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.784442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.784384    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.784903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.784875    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.785185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.784929    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.787036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.787016    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.787146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.787068    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.795563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.795535    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.796004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.795985    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgcb8\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:05.864906 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:05.864820    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:06.155303 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.155271    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"]
Nov 02 17:43:06.156340 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:06.156314    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621ff864_779d_4c0d_b079_8eff55464349.slice/crio-1b5ef4bce26fba7c9ab0a4073e9078e51a5b12cfbea4dfeb5fd087c058966f1d WatchSource:0}: Error finding container 1b5ef4bce26fba7c9ab0a4073e9078e51a5b12cfbea4dfeb5fd087c058966f1d: Status 404 returned error can't find the container with id 1b5ef4bce26fba7c9ab0a4073e9078e51a5b12cfbea4dfeb5fd087c058966f1d
Nov 02 17:43:06.158750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.158720    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:06.158815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.158783    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:06.158815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.158811    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:06.424212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.424118    5042 generic.go:356] "Generic (PLEG): container finished" podID="ab7df942-dcc0-4a93-ba77-2b1b8e86995a" containerID="ce7c0debac81e357f9aa26d2cd501773f1f36d4524024cfa4b1590866124d20a" exitCode=0
Nov 02 17:43:06.424212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.424191    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78" event={"ID":"ab7df942-dcc0-4a93-ba77-2b1b8e86995a","Type":"ContainerDied","Data":"ce7c0debac81e357f9aa26d2cd501773f1f36d4524024cfa4b1590866124d20a"}
Nov 02 17:43:06.426102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.426067    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" event={"ID":"621ff864-779d-4c0d-b079-8eff55464349","Type":"ContainerStarted","Data":"208f228e382f5dfb7e0ea2895c148fc019a7f1a7da3500b4eac7032a241b2643"}
Nov 02 17:43:06.426272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.426126    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" event={"ID":"621ff864-779d-4c0d-b079-8eff55464349","Type":"ContainerStarted","Data":"1b5ef4bce26fba7c9ab0a4073e9078e51a5b12cfbea4dfeb5fd087c058966f1d"}
Nov 02 17:43:06.427957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.427929    5042 generic.go:356] "Generic (PLEG): container finished" podID="636f30cc-655b-4e47-9d3b-c48c980db943" containerID="a3cfe1b929404a2da2340e1499795e35fbecf6d3827fe66fbd586f74a658b623" exitCode=0
Nov 02 17:43:06.428085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.428009    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf" event={"ID":"636f30cc-655b-4e47-9d3b-c48c980db943","Type":"ContainerDied","Data":"a3cfe1b929404a2da2340e1499795e35fbecf6d3827fe66fbd586f74a658b623"}
Nov 02 17:43:06.473872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.468918    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podStartSLOduration=1.468897356 podStartE2EDuration="1.468897356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:06.467424861 +0000 UTC m=+599.046834249" watchObservedRunningTime="2026-11-02 17:43:06.468897356 +0000 UTC m=+599.048306743"
Nov 02 17:43:06.693078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.693031    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"]
Nov 02 17:43:06.697544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.697518    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.741417 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.740472    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"]
Nov 02 17:43:06.807780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.805710    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6455\" (UniqueName: \"kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.818989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.811372    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.818989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.811573    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.864931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.864903    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:06.914453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.914317    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6455\" (UniqueName: \"kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.914453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.914368    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.914731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.914462    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.915256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.915229    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.917813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.917785    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.929425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.929403    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6455\" (UniqueName: \"kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455\") pod \"maas-api-revoke-sub-10430d88-hf7lb\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:06.965491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.965412    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"]
Nov 02 17:43:06.969447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.969429    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:06.999864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:06.993707    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-77f103-openshift-default-dockercfg-v9fvh\""
Nov 02 17:43:07.009090 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.008980    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"]
Nov 02 17:43:07.024221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.024005    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:07.119598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.119547    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.119956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.119929    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.120173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.120152    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.120327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.120309    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.120565 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.120549    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.120705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.120688    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.120859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.120821    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdzq\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.121003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.120986    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.130860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.121155    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226270    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226332    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdzq\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226357    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226424    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226489    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226510    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226546    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226571    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.226803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226646    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.227183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.226853    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.227183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.227167    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.227712 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.227684    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.227712 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.227695    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.228289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.228266    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.230455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.230417    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.230927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.230908    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.235231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.235184    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.236470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.236447    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdzq\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.322476 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.322435    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:07.362907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.362033    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"]
Nov 02 17:43:07.366506 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:07.365993    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbecbac2b_a694_424d_b523_17d339d879f4.slice/crio-a39dbe503d2f16fa24d8eaeb5eabb2e143df698066188abeac277f7042102185 WatchSource:0}: Error finding container a39dbe503d2f16fa24d8eaeb5eabb2e143df698066188abeac277f7042102185: Status 404 returned error can't find the container with id a39dbe503d2f16fa24d8eaeb5eabb2e143df698066188abeac277f7042102185
Nov 02 17:43:07.456954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.455603    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb" event={"ID":"becbac2b-a694-424d-b523-17d339d879f4","Type":"ContainerStarted","Data":"a39dbe503d2f16fa24d8eaeb5eabb2e143df698066188abeac277f7042102185"}
Nov 02 17:43:07.651676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.648228    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"]
Nov 02 17:43:07.662986 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:07.655964    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0586d93_c4b4_4d5c_8df3_28f87b9c259f.slice/crio-16e86be280481f88e3c61e8cc76496101c81d4ea3539fd28a62f7e4d4e62d206 WatchSource:0}: Error finding container 16e86be280481f88e3c61e8cc76496101c81d4ea3539fd28a62f7e4d4e62d206: Status 404 returned error can't find the container with id 16e86be280481f88e3c61e8cc76496101c81d4ea3539fd28a62f7e4d4e62d206
Nov 02 17:43:07.662986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.661815    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:07.662986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.661927    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:07.662986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.661971    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:07.767881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.765102    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:43:07.767881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.766054    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:43:07.770925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.770406    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:43:07.771285 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.771269    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:43:07.837899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.837255    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:07.839972 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.839833    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:07.894881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.887330    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:43:07.894881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.887387    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.961566    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5\" (UniqueName: \"kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5\") pod \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") "
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.961654    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca\") pod \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") "
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.961717    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca\") pod \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\" (UID: \"ab7df942-dcc0-4a93-ba77-2b1b8e86995a\") "
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.961805    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx\" (UniqueName: \"kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx\") pod \"636f30cc-655b-4e47-9d3b-c48c980db943\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") "
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.961890    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca\") pod \"636f30cc-655b-4e47-9d3b-c48c980db943\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") "
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.961952    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca\") pod \"636f30cc-655b-4e47-9d3b-c48c980db943\" (UID: \"636f30cc-655b-4e47-9d3b-c48c980db943\") "
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.962704    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca" pod "636f30cc-655b-4e47-9d3b-c48c980db943" (UID: "636f30cc-655b-4e47-9d3b-c48c980db943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:07.964216    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca" pod "ab7df942-dcc0-4a93-ba77-2b1b8e86995a" (UID: "ab7df942-dcc0-4a93-ba77-2b1b8e86995a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:08.014910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.007499    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca" pod "636f30cc-655b-4e47-9d3b-c48c980db943" (UID: "636f30cc-655b-4e47-9d3b-c48c980db943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:08.018421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.018368    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx" pod "636f30cc-655b-4e47-9d3b-c48c980db943" (UID: "636f30cc-655b-4e47-9d3b-c48c980db943"). InnerVolumeSpecName "kube-api-access-wwmkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:08.019891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.019591    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5" pod "ab7df942-dcc0-4a93-ba77-2b1b8e86995a" (UID: "ab7df942-dcc0-4a93-ba77-2b1b8e86995a"). InnerVolumeSpecName "kube-api-access-xtkn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:08.024603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.021515    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca" pod "ab7df942-dcc0-4a93-ba77-2b1b8e86995a" (UID: "ab7df942-dcc0-4a93-ba77-2b1b8e86995a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:08.063262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.063170    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkn5\" (UniqueName: \"kubernetes.io/projected/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-kube-api-access-xtkn5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:08.063262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.063201    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:08.063262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.063213    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7df942-dcc0-4a93-ba77-2b1b8e86995a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:08.063262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.063222    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwmkx\" (UniqueName: \"kubernetes.io/projected/636f30cc-655b-4e47-9d3b-c48c980db943-kube-api-access-wwmkx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:08.063262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.063232    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:08.063262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.063240    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636f30cc-655b-4e47-9d3b-c48c980db943-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:08.484869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.474405    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78" event={"ID":"ab7df942-dcc0-4a93-ba77-2b1b8e86995a","Type":"ContainerDied","Data":"4a7c254de0e481132759824eca7cc570e17be2d8527f9a65a5009e317e16b5c1"}
Nov 02 17:43:08.484869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.474469    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7c254de0e481132759824eca7cc570e17be2d8527f9a65a5009e317e16b5c1"
Nov 02 17:43:08.484869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.474534    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"
Nov 02 17:43:08.512862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.510476    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf" event={"ID":"636f30cc-655b-4e47-9d3b-c48c980db943","Type":"ContainerDied","Data":"826da41368ac2219938d00e3bb9f90223e85902721e1cababfd5df67dff0b077"}
Nov 02 17:43:08.512862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.510575    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="826da41368ac2219938d00e3bb9f90223e85902721e1cababfd5df67dff0b077"
Nov 02 17:43:08.512862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.510675    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"
Nov 02 17:43:08.520397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.520359    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" event={"ID":"f0586d93-c4b4-4d5c-8df3-28f87b9c259f","Type":"ContainerStarted","Data":"e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23"}
Nov 02 17:43:08.520939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.520711    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" event={"ID":"f0586d93-c4b4-4d5c-8df3-28f87b9c259f","Type":"ContainerStarted","Data":"16e86be280481f88e3c61e8cc76496101c81d4ea3539fd28a62f7e4d4e62d206"}
Nov 02 17:43:08.523953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.523747    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb" event={"ID":"becbac2b-a694-424d-b523-17d339d879f4","Type":"ContainerStarted","Data":"50d9a17b59244761e7f141792152e359334439b9513a57a0ca8b2a9d59b012f0"}
Nov 02 17:43:08.552926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.551124    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podStartSLOduration=2.551102781 podStartE2EDuration="2.551102781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:08.548976921 +0000 UTC m=+601.128386333" watchObservedRunningTime="2026-11-02 17:43:08.551102781 +0000 UTC m=+601.130512169"
Nov 02 17:43:08.614935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.614872    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb" podStartSLOduration=2.6148305670000003 podStartE2EDuration="2.614830567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:08.613586726 +0000 UTC m=+601.192996113" watchObservedRunningTime="2026-11-02 17:43:08.614830567 +0000 UTC m=+601.194239951"
Nov 02 17:43:08.867042 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.866963    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:43:08.867250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:08.867223    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 02 17:43:09.325868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.325078    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:09.325868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.325306    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:43:09.325868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.325356    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:43:09.540883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.540832    5042 generic.go:356] "Generic (PLEG): container finished" podID="becbac2b-a694-424d-b523-17d339d879f4" containerID="50d9a17b59244761e7f141792152e359334439b9513a57a0ca8b2a9d59b012f0" exitCode=0
Nov 02 17:43:09.544522 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.544493    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb" event={"ID":"becbac2b-a694-424d-b523-17d339d879f4","Type":"ContainerDied","Data":"50d9a17b59244761e7f141792152e359334439b9513a57a0ca8b2a9d59b012f0"}
Nov 02 17:43:09.876867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.866981    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:43:09.876867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:09.867056    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 02 17:43:10.328870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:10.328386    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:43:10.328870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:10.328487    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:43:10.890863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:10.877100    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:43:10.890863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:10.877162    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 02 17:43:11.058826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.058789    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:11.071252 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.071215    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455\" (UniqueName: \"kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455\") pod \"becbac2b-a694-424d-b523-17d339d879f4\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") "
Nov 02 17:43:11.071449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.071428    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca\") pod \"becbac2b-a694-424d-b523-17d339d879f4\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") "
Nov 02 17:43:11.071537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.071521    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca\") pod \"becbac2b-a694-424d-b523-17d339d879f4\" (UID: \"becbac2b-a694-424d-b523-17d339d879f4\") "
Nov 02 17:43:11.094862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.071810    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca" pod "becbac2b-a694-424d-b523-17d339d879f4" (UID: "becbac2b-a694-424d-b523-17d339d879f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:11.094862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.081424    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca" pod "becbac2b-a694-424d-b523-17d339d879f4" (UID: "becbac2b-a694-424d-b523-17d339d879f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:11.119867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.109450    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455" pod "becbac2b-a694-424d-b523-17d339d879f4" (UID: "becbac2b-a694-424d-b523-17d339d879f4"). InnerVolumeSpecName "kube-api-access-q6455". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:11.243174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.242072    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/becbac2b-a694-424d-b523-17d339d879f4-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:11.243174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.242111    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/becbac2b-a694-424d-b523-17d339d879f4-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:11.243174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.242128    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6455\" (UniqueName: \"kubernetes.io/projected/becbac2b-a694-424d-b523-17d339d879f4-kube-api-access-q6455\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:11.330871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.330262    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:43:11.330871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.330330    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:43:11.559199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.559164    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb" event={"ID":"becbac2b-a694-424d-b523-17d339d879f4","Type":"ContainerDied","Data":"a39dbe503d2f16fa24d8eaeb5eabb2e143df698066188abeac277f7042102185"}
Nov 02 17:43:11.559397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.559384    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a39dbe503d2f16fa24d8eaeb5eabb2e143df698066188abeac277f7042102185"
Nov 02 17:43:11.559481 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.559189    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"
Nov 02 17:43:11.866578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.866400    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:43:11.866578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:11.866466    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 02 17:43:12.324876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.324288    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:43:12.324876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.324358    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.350682    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"]
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351459    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ab7df942-dcc0-4a93-ba77-2b1b8e86995a" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351483    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7df942-dcc0-4a93-ba77-2b1b8e86995a" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351556    5042 cpu_manager.go:414] "containerMap: removing container" podUID="636f30cc-655b-4e47-9d3b-c48c980db943" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351565    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636f30cc-655b-4e47-9d3b-c48c980db943" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351579    5042 cpu_manager.go:414] "containerMap: removing container" podUID="becbac2b-a694-424d-b523-17d339d879f4" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351585    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="becbac2b-a694-424d-b523-17d339d879f4" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351711    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="636f30cc-655b-4e47-9d3b-c48c980db943" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351725    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="becbac2b-a694-424d-b523-17d339d879f4" containerName="revoke-keys"
Nov 02 17:43:12.355501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.351738    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7df942-dcc0-4a93-ba77-2b1b8e86995a" containerName="revoke-keys"
Nov 02 17:43:12.365885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.358088    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.366260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.366237    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:43:12.366558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.366541    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:43:12.370285 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.370257    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"]
Nov 02 17:43:12.370742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.370703    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:43:12.370960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.370927    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:12.467864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.464493    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7hx\" (UniqueName: \"kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.467864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.464617    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.467864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.464658    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.577925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.572195    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7hx\" (UniqueName: \"kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.577925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.572336    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.577925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.572378    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.623865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.614865    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.655759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.655720    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7hx\" (UniqueName: \"kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.657535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.657495    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdc3543f-dffj4\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.722921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.720611    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:12.866054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.865551    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:43:12.866054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:12.865607    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 02 17:43:13.036280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:13.034959    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"]
Nov 02 17:43:13.044000 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:13.043958    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb86630_8d96_4cbe_9eb4_2b6f9fee96ac.slice/crio-89905823551af951b045b3b0dfefcc7be562f6cd346d39b1ab1670564f0a8178 WatchSource:0}: Error finding container 89905823551af951b045b3b0dfefcc7be562f6cd346d39b1ab1670564f0a8178: Status 404 returned error can't find the container with id 89905823551af951b045b3b0dfefcc7be562f6cd346d39b1ab1670564f0a8178
Nov 02 17:43:13.323211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:13.323096    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:43:13.323211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:13.323188    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:43:13.670618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:13.670523    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4" event={"ID":"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac","Type":"ContainerStarted","Data":"d613585d04d202f0d760a78d332c0e9d6290bd9f4c32dfd74403aeae904f29a8"}
Nov 02 17:43:13.670618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:13.670579    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4" event={"ID":"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac","Type":"ContainerStarted","Data":"89905823551af951b045b3b0dfefcc7be562f6cd346d39b1ab1670564f0a8178"}
Nov 02 17:43:13.705322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:13.705255    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4" podStartSLOduration=1.705232246 podStartE2EDuration="1.705232246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:13.694256323 +0000 UTC m=+606.273665723" watchObservedRunningTime="2026-11-02 17:43:13.705232246 +0000 UTC m=+606.284641633"
Nov 02 17:43:14.323621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.323581    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 02 17:43:14.323808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.323642    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 02 17:43:14.359664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.359632    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:14.360098 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.360077    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:14.361500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.361462    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:43:14.680770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.680732    5042 generic.go:356] "Generic (PLEG): container finished" podID="0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" containerID="d613585d04d202f0d760a78d332c0e9d6290bd9f4c32dfd74403aeae904f29a8" exitCode=0
Nov 02 17:43:14.681228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:14.680830    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4" event={"ID":"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac","Type":"ContainerDied","Data":"d613585d04d202f0d760a78d332c0e9d6290bd9f4c32dfd74403aeae904f29a8"}
Nov 02 17:43:15.328583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:15.328546    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:15.329021 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:15.329000    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:15.329962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:15.329941    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 17:43:16.104881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.101726    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:16.166997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.166960    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx\" (UniqueName: \"kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx\") pod \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") "
Nov 02 17:43:16.167186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.167010    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca\") pod \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") "
Nov 02 17:43:16.167186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.167116    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca\") pod \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\" (UID: \"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac\") "
Nov 02 17:43:16.167470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.167439    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca" pod "0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" (UID: "0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:16.167579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.167557    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:16.169791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.169758    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx" pod "0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" (UID: "0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac"). InnerVolumeSpecName "kube-api-access-cw7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:16.169960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.169914    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca" pod "0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" (UID: "0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:16.268511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.268475    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw7hx\" (UniqueName: \"kubernetes.io/projected/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-kube-api-access-cw7hx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:16.268511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.268511    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:16.291195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.291108    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"]
Nov 02 17:43:16.291974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.291953    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" containerName="revoke-keys"
Nov 02 17:43:16.292033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.291976    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" containerName="revoke-keys"
Nov 02 17:43:16.292129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.292117    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" containerName="revoke-keys"
Nov 02 17:43:16.296668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.296650    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.301998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.301971    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"]
Nov 02 17:43:16.369189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.369152    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.369370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.369229    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twt6m\" (UniqueName: \"kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.369370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.369333    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.470082 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.470045    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.470257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.470123    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twt6m\" (UniqueName: \"kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.470257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.470155    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.471053 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.470978    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.472803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.472771    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.479714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.479649    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twt6m\" (UniqueName: \"kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m\") pod \"maas-api-revoke-sub-d9c47287-rvjnj\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.616414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.616321    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:16.701260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.701228    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"
Nov 02 17:43:16.701423 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.701335    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4" event={"ID":"0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac","Type":"ContainerDied","Data":"89905823551af951b045b3b0dfefcc7be562f6cd346d39b1ab1670564f0a8178"}
Nov 02 17:43:16.701423 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.701374    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89905823551af951b045b3b0dfefcc7be562f6cd346d39b1ab1670564f0a8178"
Nov 02 17:43:16.897122 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:16.897088    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"]
Nov 02 17:43:16.897726 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:16.897699    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012ec9b3_5eed_4bbd_a0fb_54144a80e8d8.slice/crio-cafec9812f6c8211becc4d91c0d5f92483bab219338fb84dd1112e7c1090583b WatchSource:0}: Error finding container cafec9812f6c8211becc4d91c0d5f92483bab219338fb84dd1112e7c1090583b: Status 404 returned error can't find the container with id cafec9812f6c8211becc4d91c0d5f92483bab219338fb84dd1112e7c1090583b
Nov 02 17:43:17.709536 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:17.709491    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj" event={"ID":"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8","Type":"ContainerStarted","Data":"e1663ce34e9c9679e4ed11a503f66d16d398132775f150a13c332ec47afb6f55"}
Nov 02 17:43:17.709978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:17.709554    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj" event={"ID":"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8","Type":"ContainerStarted","Data":"cafec9812f6c8211becc4d91c0d5f92483bab219338fb84dd1112e7c1090583b"}
Nov 02 17:43:17.729591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:17.729535    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj" podStartSLOduration=1.729519266 podStartE2EDuration="1.729519266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:17.726974297 +0000 UTC m=+610.306383683" watchObservedRunningTime="2026-11-02 17:43:17.729519266 +0000 UTC m=+610.308928652"
Nov 02 17:43:18.025796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.025711    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"]
Nov 02 17:43:18.029897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.029877    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.039873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.039722    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"]
Nov 02 17:43:18.091961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.091904    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.092157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.092013    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhcdx\" (UniqueName: \"kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.092157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.092064    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.193339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.193300    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhcdx\" (UniqueName: \"kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.193514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.193357    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.193514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.193494    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.194280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.194179    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.196147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.196119    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.203070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.203044    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhcdx\" (UniqueName: \"kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx\") pod \"maas-api-revoke-sub-a41480aa-j4d8b\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.360540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.360463    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:18.541304 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:43:18.541265    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:43:18.648211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.648175    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"]
Nov 02 17:43:18.649757 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:18.649728    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd775878_72e3_43d4_8a09_562005933027.slice/crio-ed82d040f966aa6d55b503c0b0eb02bed384a3e9ddf4befb9b0e6f5dd95cfc9a WatchSource:0}: Error finding container ed82d040f966aa6d55b503c0b0eb02bed384a3e9ddf4befb9b0e6f5dd95cfc9a: Status 404 returned error can't find the container with id ed82d040f966aa6d55b503c0b0eb02bed384a3e9ddf4befb9b0e6f5dd95cfc9a
Nov 02 17:43:18.717451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.717418    5042 generic.go:356] "Generic (PLEG): container finished" podID="012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" containerID="e1663ce34e9c9679e4ed11a503f66d16d398132775f150a13c332ec47afb6f55" exitCode=0
Nov 02 17:43:18.717975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.717499    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj" event={"ID":"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8","Type":"ContainerDied","Data":"e1663ce34e9c9679e4ed11a503f66d16d398132775f150a13c332ec47afb6f55"}
Nov 02 17:43:18.719055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:18.719015    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b" event={"ID":"fd775878-72e3-43d4-8a09-562005933027","Type":"ContainerStarted","Data":"ed82d040f966aa6d55b503c0b0eb02bed384a3e9ddf4befb9b0e6f5dd95cfc9a"}
Nov 02 17:43:19.727349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.727315    5042 generic.go:356] "Generic (PLEG): container finished" podID="fd775878-72e3-43d4-8a09-562005933027" containerID="28bbcd0e0852cc14340b74019a73fbc08c3d9ea9ac3c150cc0f5a363b13c5e05" exitCode=0
Nov 02 17:43:19.727790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.727445    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b" event={"ID":"fd775878-72e3-43d4-8a09-562005933027","Type":"ContainerDied","Data":"28bbcd0e0852cc14340b74019a73fbc08c3d9ea9ac3c150cc0f5a363b13c5e05"}
Nov 02 17:43:19.883575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.883539    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:19.921790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.921746    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca\") pod \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") "
Nov 02 17:43:19.922032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.921817    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca\") pod \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") "
Nov 02 17:43:19.922032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.921937    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m\" (UniqueName: \"kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m\") pod \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\" (UID: \"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8\") "
Nov 02 17:43:19.922241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.922216    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca" pod "012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" (UID: "012ec9b3-5eed-4bbd-a0fb-54144a80e8d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:19.922471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.922451    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:19.924302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.924273    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca" pod "012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" (UID: "012ec9b3-5eed-4bbd-a0fb-54144a80e8d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:19.924453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:19.924428    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m" pod "012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" (UID: "012ec9b3-5eed-4bbd-a0fb-54144a80e8d8"). InnerVolumeSpecName "kube-api-access-twt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:20.023199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:20.023160    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:20.023199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:20.023191    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twt6m\" (UniqueName: \"kubernetes.io/projected/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8-kube-api-access-twt6m\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:20.739516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:20.739484    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"
Nov 02 17:43:20.741459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:20.739790    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj" event={"ID":"012ec9b3-5eed-4bbd-a0fb-54144a80e8d8","Type":"ContainerDied","Data":"cafec9812f6c8211becc4d91c0d5f92483bab219338fb84dd1112e7c1090583b"}
Nov 02 17:43:20.741645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:20.741628    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cafec9812f6c8211becc4d91c0d5f92483bab219338fb84dd1112e7c1090583b"
Nov 02 17:43:20.921763 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:20.921734    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:21.040923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.040298    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca\") pod \"fd775878-72e3-43d4-8a09-562005933027\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") "
Nov 02 17:43:21.040923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.040376    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx\" (UniqueName: \"kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx\") pod \"fd775878-72e3-43d4-8a09-562005933027\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") "
Nov 02 17:43:21.040923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.040529    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca\") pod \"fd775878-72e3-43d4-8a09-562005933027\" (UID: \"fd775878-72e3-43d4-8a09-562005933027\") "
Nov 02 17:43:21.060110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.041226    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca" pod "fd775878-72e3-43d4-8a09-562005933027" (UID: "fd775878-72e3-43d4-8a09-562005933027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:21.060110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.053720    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca" pod "fd775878-72e3-43d4-8a09-562005933027" (UID: "fd775878-72e3-43d4-8a09-562005933027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:21.071907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.067551    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx" pod "fd775878-72e3-43d4-8a09-562005933027" (UID: "fd775878-72e3-43d4-8a09-562005933027"). InnerVolumeSpecName "kube-api-access-jhcdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:21.142526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.142484    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd775878-72e3-43d4-8a09-562005933027-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:21.142526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.142528    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd775878-72e3-43d4-8a09-562005933027-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:21.142798 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.142546    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhcdx\" (UniqueName: \"kubernetes.io/projected/fd775878-72e3-43d4-8a09-562005933027-kube-api-access-jhcdx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:21.748558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.748527    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"
Nov 02 17:43:21.748986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.748520    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b" event={"ID":"fd775878-72e3-43d4-8a09-562005933027","Type":"ContainerDied","Data":"ed82d040f966aa6d55b503c0b0eb02bed384a3e9ddf4befb9b0e6f5dd95cfc9a"}
Nov 02 17:43:21.748986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:21.748665    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed82d040f966aa6d55b503c0b0eb02bed384a3e9ddf4befb9b0e6f5dd95cfc9a"
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.244427    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"]
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.244993    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fd775878-72e3-43d4-8a09-562005933027" containerName="revoke-keys"
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.245006    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd775878-72e3-43d4-8a09-562005933027" containerName="revoke-keys"
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.245017    5042 cpu_manager.go:414] "containerMap: removing container" podUID="012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" containerName="revoke-keys"
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.245023    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" containerName="revoke-keys"
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.245102    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd775878-72e3-43d4-8a09-562005933027" containerName="revoke-keys"
Nov 02 17:43:28.246281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.245110    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" containerName="revoke-keys"
Nov 02 17:43:28.249592 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.249567    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.254387 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.254359    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:43:28.254581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.254486    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:28.254581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.254554    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:43:28.255245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.255222    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:43:28.259087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.258927    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"]
Nov 02 17:43:28.403608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.403556    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfqx6\" (UniqueName: \"kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.414128 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.414087    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.414587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.414563    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.521710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.521608    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.522922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.522895    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfqx6\" (UniqueName: \"kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.523493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.523469    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.524096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.522742    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.538160 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.538132    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.542357 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.542330    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfqx6\" (UniqueName: \"kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6\") pod \"maas-api-revoke-sub-2e9ecf52-lj5nq\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.568886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.568826    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:28.961548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:28.961520    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"]
Nov 02 17:43:28.963040 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:28.963008    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764ed14c_791a_4f72_bffc_3a60e6f65235.slice/crio-bba347cedcbea1bb03bebeda3d2c9ba3840fdb24f7632b56d09d306b9ce0efe1 WatchSource:0}: Error finding container bba347cedcbea1bb03bebeda3d2c9ba3840fdb24f7632b56d09d306b9ce0efe1: Status 404 returned error can't find the container with id bba347cedcbea1bb03bebeda3d2c9ba3840fdb24f7632b56d09d306b9ce0efe1
Nov 02 17:43:29.861072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:29.861028    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq" event={"ID":"764ed14c-791a-4f72-bffc-3a60e6f65235","Type":"ContainerStarted","Data":"e808f16e7200a783e940489b559250fc64704dfc49e52ac3517c40c5a964da97"}
Nov 02 17:43:29.861588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:29.861104    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq" event={"ID":"764ed14c-791a-4f72-bffc-3a60e6f65235","Type":"ContainerStarted","Data":"bba347cedcbea1bb03bebeda3d2c9ba3840fdb24f7632b56d09d306b9ce0efe1"}
Nov 02 17:43:29.882454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:29.882389    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq" podStartSLOduration=1.882369911 podStartE2EDuration="1.882369911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:29.878444774 +0000 UTC m=+622.457854161" watchObservedRunningTime="2026-11-02 17:43:29.882369911 +0000 UTC m=+622.461779498"
Nov 02 17:43:30.868515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:30.868478    5042 generic.go:356] "Generic (PLEG): container finished" podID="764ed14c-791a-4f72-bffc-3a60e6f65235" containerID="e808f16e7200a783e940489b559250fc64704dfc49e52ac3517c40c5a964da97" exitCode=0
Nov 02 17:43:30.868940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:30.868560    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq" event={"ID":"764ed14c-791a-4f72-bffc-3a60e6f65235","Type":"ContainerDied","Data":"e808f16e7200a783e940489b559250fc64704dfc49e52ac3517c40c5a964da97"}
Nov 02 17:43:32.039774 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.039743    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:32.090472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.090438    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca\") pod \"764ed14c-791a-4f72-bffc-3a60e6f65235\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") "
Nov 02 17:43:32.090655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.090521    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6\" (UniqueName: \"kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6\") pod \"764ed14c-791a-4f72-bffc-3a60e6f65235\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") "
Nov 02 17:43:32.090655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.090557    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca\") pod \"764ed14c-791a-4f72-bffc-3a60e6f65235\" (UID: \"764ed14c-791a-4f72-bffc-3a60e6f65235\") "
Nov 02 17:43:32.091062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.091018    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca" pod "764ed14c-791a-4f72-bffc-3a60e6f65235" (UID: "764ed14c-791a-4f72-bffc-3a60e6f65235"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:32.093157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.093122    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca" pod "764ed14c-791a-4f72-bffc-3a60e6f65235" (UID: "764ed14c-791a-4f72-bffc-3a60e6f65235"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:32.093267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.093196    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6" pod "764ed14c-791a-4f72-bffc-3a60e6f65235" (UID: "764ed14c-791a-4f72-bffc-3a60e6f65235"). InnerVolumeSpecName "kube-api-access-hfqx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:32.192172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.192133    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfqx6\" (UniqueName: \"kubernetes.io/projected/764ed14c-791a-4f72-bffc-3a60e6f65235-kube-api-access-hfqx6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:32.192349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.192175    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:32.192349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.192193    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764ed14c-791a-4f72-bffc-3a60e6f65235-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:43:32.885211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.885181    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"
Nov 02 17:43:32.885418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.885173    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq" event={"ID":"764ed14c-791a-4f72-bffc-3a60e6f65235","Type":"ContainerDied","Data":"bba347cedcbea1bb03bebeda3d2c9ba3840fdb24f7632b56d09d306b9ce0efe1"}
Nov 02 17:43:32.885418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:32.885322    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bba347cedcbea1bb03bebeda3d2c9ba3840fdb24f7632b56d09d306b9ce0efe1"
Nov 02 17:43:42.171737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.171699    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"]
Nov 02 17:43:42.173828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.173665    5042 cpu_manager.go:414] "containerMap: removing container" podUID="764ed14c-791a-4f72-bffc-3a60e6f65235" containerName="revoke-keys"
Nov 02 17:43:42.174197 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.174150    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764ed14c-791a-4f72-bffc-3a60e6f65235" containerName="revoke-keys"
Nov 02 17:43:42.174499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.174481    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="764ed14c-791a-4f72-bffc-3a60e6f65235" containerName="revoke-keys"
Nov 02 17:43:42.182675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.182639    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"]
Nov 02 17:43:42.183065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.183045    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.188891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.188868    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 17:43:42.196206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.196181    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3cb9c46838e9\""
Nov 02 17:43:42.212010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.198102    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 17:43:42.212209 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.198199    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 17:43:42.212209 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.198256    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 17:43:42.212209 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.198381    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4kxjc\""
Nov 02 17:43:42.272508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.272477    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.272705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.272647    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.272773 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.272706    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.272773 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.272744    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7qlm\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.374011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.373975    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.374228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.374048    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.374228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.374079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7qlm\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.374228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.374203    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.375027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.374999    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.375394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.375084    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.377016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.376993    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.383574 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.383550    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7qlm\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm\") pod \"payload-processing-external-model-6f59d4c79f-g25tq\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") " pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.522489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.522446    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:42.815195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:42.815164    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"]
Nov 02 17:43:42.816521 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:42.816492    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f32a1df_c389_497f_99c6_37d256d45d70.slice/crio-354b398a93868c6c09a14430461d28c68c6ff7eeb71f125874355d66753137e6 WatchSource:0}: Error finding container 354b398a93868c6c09a14430461d28c68c6ff7eeb71f125874355d66753137e6: Status 404 returned error can't find the container with id 354b398a93868c6c09a14430461d28c68c6ff7eeb71f125874355d66753137e6
Nov 02 17:43:43.000316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:43.000275    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq" event={"ID":"7f32a1df-c389-497f-99c6-37d256d45d70","Type":"ContainerStarted","Data":"aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6"}
Nov 02 17:43:43.000519 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:43.000341    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq" event={"ID":"7f32a1df-c389-497f-99c6-37d256d45d70","Type":"ContainerStarted","Data":"354b398a93868c6c09a14430461d28c68c6ff7eeb71f125874355d66753137e6"}
Nov 02 17:43:43.000519 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:43.000419    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:43.021701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:43.021639    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq" podStartSLOduration=1.021620853 podStartE2EDuration="1.021620853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:43.020029043 +0000 UTC m=+635.599438429" watchObservedRunningTime="2026-11-02 17:43:43.021620853 +0000 UTC m=+635.601030238"
Nov 02 17:43:48.543064 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:43:48.543015    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:43:52.223020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.222975    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"]
Nov 02 17:43:52.227135 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.227091    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.238457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.238432    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"]
Nov 02 17:43:52.255695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.255660    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"]
Nov 02 17:43:52.263159 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.263131    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.278444 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.278315    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"]
Nov 02 17:43:52.305027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.304994    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305039    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7s7b\" (UniqueName: \"kubernetes.io/projected/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-kube-api-access-j7s7b\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305060    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305187    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305244    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305277    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305382    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305455    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305491    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305612 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305593    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305673 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305628    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305676    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mhhc\" (UniqueName: \"kubernetes.io/projected/5a334a6a-1a82-435e-9785-1047ec1c3487-kube-api-access-2mhhc\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305748    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305799    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.305892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305825    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305865    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305903    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a334a6a-1a82-435e-9785-1047ec1c3487-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.305974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.305941    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.406979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.406942    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.406989    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407010    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a334a6a-1a82-435e-9785-1047ec1c3487-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407051    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407102    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407143    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s7b\" (UniqueName: \"kubernetes.io/projected/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-kube-api-access-j7s7b\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407168    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407211    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407261    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407291    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407340    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407383    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407412    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407413    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407455    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407484    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407520    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407520    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407802    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407533    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mhhc\" (UniqueName: \"kubernetes.io/projected/5a334a6a-1a82-435e-9785-1047ec1c3487-kube-api-access-2mhhc\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407899    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407956    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407950    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.407984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.407964    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a334a6a-1a82-435e-9785-1047ec1c3487-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.408462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.408270    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.408680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.408654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.408806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.408747    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.409005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.408982    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.410649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.410623    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.410765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.410663    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.410829 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.410815    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.411054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.411034    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.423888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.423828    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a334a6a-1a82-435e-9785-1047ec1c3487-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.424157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.424131    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.425010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.424987    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s7b\" (UniqueName: \"kubernetes.io/projected/bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11-kube-api-access-j7s7b\") pod \"maas-default-gateway-openshift-default-65c448b547-xj2fs\" (UID: \"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.425625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.425606    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mhhc\" (UniqueName: \"kubernetes.io/projected/5a334a6a-1a82-435e-9785-1047ec1c3487-kube-api-access-2mhhc\") pod \"maas-default-gateway-openshift-default-65c448b547-dtrzt\" (UID: \"5a334a6a-1a82-435e-9785-1047ec1c3487\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.543878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.543788    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:52.585268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.583941    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:52.846958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.846933    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"]
Nov 02 17:43:52.847624 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:52.847600    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1d62b5_27ee_4e99_9cb7_5ee8caebfd11.slice/crio-9c618caae51f5788c5e145d3d612338dff3d1365599b3fc7a118763a0a4e16a9 WatchSource:0}: Error finding container 9c618caae51f5788c5e145d3d612338dff3d1365599b3fc7a118763a0a4e16a9: Status 404 returned error can't find the container with id 9c618caae51f5788c5e145d3d612338dff3d1365599b3fc7a118763a0a4e16a9
Nov 02 17:43:52.850257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.850225    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:52.850332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.850288    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:52.850332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.850314    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:52.897807 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.897783    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"]
Nov 02 17:43:52.898619 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:52.898594    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a334a6a_1a82_435e_9785_1047ec1c3487.slice/crio-b48be960f58c2f7891b77ca9a2cc3ea04801657f4a615b597208f1e057133914 WatchSource:0}: Error finding container b48be960f58c2f7891b77ca9a2cc3ea04801657f4a615b597208f1e057133914: Status 404 returned error can't find the container with id b48be960f58c2f7891b77ca9a2cc3ea04801657f4a615b597208f1e057133914
Nov 02 17:43:52.902091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.902055    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:52.902199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.902143    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:52.902199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:52.902186    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:53.077291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.075977    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" event={"ID":"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11","Type":"ContainerStarted","Data":"ac8805cb461750e97d81bebc42be1447787911f527f2f073f24e3f324db0da5c"}
Nov 02 17:43:53.077291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.076042    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" event={"ID":"bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11","Type":"ContainerStarted","Data":"9c618caae51f5788c5e145d3d612338dff3d1365599b3fc7a118763a0a4e16a9"}
Nov 02 17:43:53.097617 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.097473    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" event={"ID":"5a334a6a-1a82-435e-9785-1047ec1c3487","Type":"ContainerStarted","Data":"64005eeb3dbcb0881f54b3533d5a3cee4dfe693bb5d17500ac550ba690c4b236"}
Nov 02 17:43:53.097617 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.097553    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" event={"ID":"5a334a6a-1a82-435e-9785-1047ec1c3487","Type":"ContainerStarted","Data":"b48be960f58c2f7891b77ca9a2cc3ea04801657f4a615b597208f1e057133914"}
Nov 02 17:43:53.107746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.107689    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podStartSLOduration=1.107670535 podStartE2EDuration="1.107670535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:53.103891788 +0000 UTC m=+645.683301176" watchObservedRunningTime="2026-11-02 17:43:53.107670535 +0000 UTC m=+645.687079921"
Nov 02 17:43:53.131861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.131501    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podStartSLOduration=1.131486147 podStartE2EDuration="1.131486147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:53.130415242 +0000 UTC m=+645.709824623" watchObservedRunningTime="2026-11-02 17:43:53.131486147 +0000 UTC m=+645.710895536"
Nov 02 17:43:53.544355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.544325    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:43:53.546351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.546323    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:53.546467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.546369    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:53.584343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.584302    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:43:53.585942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.585913    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:53.586092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:53.585962    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:54.027093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:54.027055    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:43:54.544752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:54.544714    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:54.545191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:54.544777    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:54.586355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:54.586316    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:54.586646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:54.586619    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:55.544609 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:55.544573    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:55.544821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:55.544631    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:55.588864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:55.588260    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:55.588864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:55.588325    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:56.550172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:56.550134    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:56.551268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:56.550764    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:56.589322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:56.589282    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:56.589645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:56.589612    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:57.578867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:57.570968    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:57.578867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:57.571028    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:57.614868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:57.612223    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:57.614868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:57.612286    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:58.544714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:58.544681    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:58.544926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:58.544751    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:58.593477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:58.593442    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:58.594073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:58.594039    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:58.949532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:58.947914    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"]
Nov 02 17:43:59.061950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.061899    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"]
Nov 02 17:43:59.062366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.062351    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.123868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.108789    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0c3a47e60ef3\""
Nov 02 17:43:59.123868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.110572    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjkdb\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.123868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.110630    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.123868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.110705    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.123868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.110741    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.239953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.234590    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjkdb\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.240230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.240204    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.242859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.241530    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.242859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.241576    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.242859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.242640    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.242859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.242749    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.246148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.246126    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjkdb\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.246784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.246763    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials\") pod \"payload-processing-external-model-bb6976b77-7rk7g\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") " pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.458763 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.458725    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:43:59.545353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.545009    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:43:59.545353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.545057    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:43:59.585914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.585451    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:59.585914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.585512    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:43:59.781238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:43:59.780786    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"]
Nov 02 17:43:59.783363 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:43:59.783328    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98652732_484b_4eeb_8292_9a0d4be2f713.slice/crio-d97096b58c954888f411afc390794571bafee7d9d790fcab784559351bff8197 WatchSource:0}: Error finding container d97096b58c954888f411afc390794571bafee7d9d790fcab784559351bff8197: Status 404 returned error can't find the container with id d97096b58c954888f411afc390794571bafee7d9d790fcab784559351bff8197
Nov 02 17:44:00.258997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.258958    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g" event={"ID":"98652732-484b-4eeb-8292-9a0d4be2f713","Type":"ContainerStarted","Data":"d275f576f48125a31057b6d4394a392b3108313ae6212a15056700e0e973ea40"}
Nov 02 17:44:00.259183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.259022    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g" event={"ID":"98652732-484b-4eeb-8292-9a0d4be2f713","Type":"ContainerStarted","Data":"d97096b58c954888f411afc390794571bafee7d9d790fcab784559351bff8197"}
Nov 02 17:44:00.259183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.259060    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:44:00.320131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.320060    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g" podStartSLOduration=2.320036816 podStartE2EDuration="2.320036816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:00.295258942 +0000 UTC m=+652.874668328" watchObservedRunningTime="2026-11-02 17:44:00.320036816 +0000 UTC m=+652.899446203"
Nov 02 17:44:00.544987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.544893    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Nov 02 17:44:00.544987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.544950    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Nov 02 17:44:00.585090 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.585050    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:44:00.585295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:00.585120    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Nov 02 17:44:02.317866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:02.316866    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:44:02.318415 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:02.318384    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:44:02.319355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:02.319334    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt"
Nov 02 17:44:02.487155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:02.487120    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:44:02.487892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:02.487871    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:44:02.488742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:02.488721    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs"
Nov 02 17:44:11.299268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:11.298048    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:44:11.522894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:11.521251    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"]
Nov 02 17:44:12.028879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.027146    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:12.037866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.030063    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:12.088866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.052009    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:12.246874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.212483    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:12.246874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.212988    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:44:12.266866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.251442    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:44:12.310926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:12.289713    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:16.522655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.522605    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq" podUID="7f32a1df-c389-497f-99c6-37d256d45d70" containerName="payload-processing" containerID="cri-o://aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6" gracePeriod=25
Nov 02 17:44:16.820588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.820559    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:44:16.885000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.883877    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume\") pod \"7f32a1df-c389-497f-99c6-37d256d45d70\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") "
Nov 02 17:44:16.885000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.884034    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume\") pod \"7f32a1df-c389-497f-99c6-37d256d45d70\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") "
Nov 02 17:44:16.885000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.884088    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm\") pod \"7f32a1df-c389-497f-99c6-37d256d45d70\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") "
Nov 02 17:44:16.885000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.884156    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials\") pod \"7f32a1df-c389-497f-99c6-37d256d45d70\" (UID: \"7f32a1df-c389-497f-99c6-37d256d45d70\") "
Nov 02 17:44:16.886066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.886038    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume" pod "7f32a1df-c389-497f-99c6-37d256d45d70" (UID: "7f32a1df-c389-497f-99c6-37d256d45d70"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:16.888374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.888351    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume" pod "7f32a1df-c389-497f-99c6-37d256d45d70" (UID: "7f32a1df-c389-497f-99c6-37d256d45d70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:16.892007 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.891947    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials" pod "7f32a1df-c389-497f-99c6-37d256d45d70" (UID: "7f32a1df-c389-497f-99c6-37d256d45d70"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:16.899047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.899010    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm" pod "7f32a1df-c389-497f-99c6-37d256d45d70" (UID: "7f32a1df-c389-497f-99c6-37d256d45d70"). InnerVolumeSpecName "kube-api-access-s7qlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:16.985903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.985864    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:16.985903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.985896    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7qlm\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-kube-api-access-s7qlm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:16.985903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.985905    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f32a1df-c389-497f-99c6-37d256d45d70-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:16.986147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:16.985914    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f32a1df-c389-497f-99c6-37d256d45d70-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:17.560295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.560264    5042 generic.go:356] "Generic (PLEG): container finished" podID="7f32a1df-c389-497f-99c6-37d256d45d70" containerID="aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6" exitCode=0
Nov 02 17:44:17.560706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.560311    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"
Nov 02 17:44:17.560706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.560348    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq" event={"ID":"7f32a1df-c389-497f-99c6-37d256d45d70","Type":"ContainerDied","Data":"aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6"}
Nov 02 17:44:17.560706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.560401    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq" event={"ID":"7f32a1df-c389-497f-99c6-37d256d45d70","Type":"ContainerDied","Data":"354b398a93868c6c09a14430461d28c68c6ff7eeb71f125874355d66753137e6"}
Nov 02 17:44:17.560706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.560427    5042 scope.go:122] "RemoveContainer" containerID="aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6"
Nov 02 17:44:17.571208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.571187    5042 scope.go:122] "RemoveContainer" containerID="aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6"
Nov 02 17:44:17.571538 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:44:17.571519    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6\": container with ID starting with aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6 not found: ID does not exist" containerID="aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6"
Nov 02 17:44:17.571625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.571546    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6"} err="failed to get container status \"aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6\": rpc error: code = NotFound desc = could not find container \"aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6\": container with ID starting with aee44a6e3eb8b10d840346aa27226a8864af18dd4ebc18c61c4e967ef348f5f6 not found: ID does not exist"
Nov 02 17:44:17.583695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.583658    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"]
Nov 02 17:44:17.591389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:17.591348    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f59d4c79f-g25tq"]
Nov 02 17:44:18.540451 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:44:18.540404    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:44:19.543293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:19.543260    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f32a1df-c389-497f-99c6-37d256d45d70" path="/var/lib/kubelet/pods/7f32a1df-c389-497f-99c6-37d256d45d70/volumes"
Nov 02 17:44:20.222770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.222734    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"]
Nov 02 17:44:20.223814 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.223679    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7f32a1df-c389-497f-99c6-37d256d45d70" containerName="payload-processing"
Nov 02 17:44:20.223814 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.223703    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f32a1df-c389-497f-99c6-37d256d45d70" containerName="payload-processing"
Nov 02 17:44:20.224071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.223868    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f32a1df-c389-497f-99c6-37d256d45d70" containerName="payload-processing"
Nov 02 17:44:20.227351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.227326    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.237310 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.236487    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-66eed1\"/\"distinct-2-w2-66eed1-kserve-self-signed-certs\""
Nov 02 17:44:20.242602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.242571    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"]
Nov 02 17:44:20.328129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.328097    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.328129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.328137    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.328360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.328272    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.328360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.328332    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.328434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.328372    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgdn\" (UniqueName: \"kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.328540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.328511    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.430294    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.430387    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.430436    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgdn\" (UniqueName: \"kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.430522    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.430592    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.430625    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.431107    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.431766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.431363    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.432744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.432717    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.435305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.435276    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.436100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.436049    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.450170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.450045    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgdn\" (UniqueName: \"kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn\") pod \"distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") " pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.544880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.544769    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:20.668676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.668637    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"]
Nov 02 17:44:20.674672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.674629    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.687428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.686104    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"]
Nov 02 17:44:20.734047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734010    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734055    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734083    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734103    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734249    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734301    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734434    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734472    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.734602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.734505    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qrcm\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835295    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835338    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835372    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835393    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835461    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835516    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835646    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835691    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835734    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qrcm\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.835967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.835765    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.836570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.836542    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.841891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.836587    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.841891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.837657    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.841891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.837926    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.851248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.851213    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.851433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.851213    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.856926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.854658    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.856926 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:44:20.855271    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9126f0ac_686e_4dce_8309_31c7cb08350c.slice/crio-eb18ccf2e68ff58b2ca225fb83d1ea17b5c527e44e9f4a4e4183fc069c469b81 WatchSource:0}: Error finding container eb18ccf2e68ff58b2ca225fb83d1ea17b5c527e44e9f4a4e4183fc069c469b81: Status 404 returned error can't find the container with id eb18ccf2e68ff58b2ca225fb83d1ea17b5c527e44e9f4a4e4183fc069c469b81
Nov 02 17:44:20.856926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.856328    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qrcm\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:20.857549 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.857523    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"]
Nov 02 17:44:20.999286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:20.999249    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:21.287603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.287563    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"]
Nov 02 17:44:21.288781 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:44:21.288747    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d03ddb_5a13_4516_bdbf_8229f0caec89.slice/crio-3064864cc7b6d5553a80f0e0988393c25af84ea6433d6f9eadcafbafa4c09321 WatchSource:0}: Error finding container 3064864cc7b6d5553a80f0e0988393c25af84ea6433d6f9eadcafbafa4c09321: Status 404 returned error can't find the container with id 3064864cc7b6d5553a80f0e0988393c25af84ea6433d6f9eadcafbafa4c09321
Nov 02 17:44:21.291689 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.291650    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:21.291773 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.291735    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:21.291816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.291777    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:21.602169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.597241    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" event={"ID":"b9d03ddb-5a13-4516-bdbf-8229f0caec89","Type":"ContainerStarted","Data":"110fe61ad703b6513d309023bc8955ced31d09ee27bc2408e6bc438e5106986f"}
Nov 02 17:44:21.602169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.597299    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" event={"ID":"b9d03ddb-5a13-4516-bdbf-8229f0caec89","Type":"ContainerStarted","Data":"3064864cc7b6d5553a80f0e0988393c25af84ea6433d6f9eadcafbafa4c09321"}
Nov 02 17:44:21.602745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.602612    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" event={"ID":"9126f0ac-686e-4dce-8309-31c7cb08350c","Type":"ContainerStarted","Data":"47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28"}
Nov 02 17:44:21.602745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.602658    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" event={"ID":"9126f0ac-686e-4dce-8309-31c7cb08350c","Type":"ContainerStarted","Data":"eb18ccf2e68ff58b2ca225fb83d1ea17b5c527e44e9f4a4e4183fc069c469b81"}
Nov 02 17:44:21.623288 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.623231    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podStartSLOduration=1.62321239 podStartE2EDuration="1.62321239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:21.622203262 +0000 UTC m=+674.201612649" watchObservedRunningTime="2026-11-02 17:44:21.62321239 +0000 UTC m=+674.202621777"
Nov 02 17:44:21.999935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:21.999894    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:22.545664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:22.545630    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:23.007014 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:23.006967    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 17:44:23.007532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:23.007052    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 17:44:24.000498 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:24.000453    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 17:44:24.000709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:24.000524    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 17:44:25.003028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:25.002951    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 17:44:25.003028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:25.003017    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 17:44:26.003339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:26.003298    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 17:44:26.004005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:26.003963    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 17:44:27.003517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.003185    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 17:44:27.003517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.003235    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 17:44:27.664383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.664323    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podStartSLOduration=7.664308061 podStartE2EDuration="7.664308061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:21.646068937 +0000 UTC m=+674.225478323" watchObservedRunningTime="2026-11-02 17:44:27.664308061 +0000 UTC m=+680.243717448"
Nov 02 17:44:27.665242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.665217    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"]
Nov 02 17:44:27.670402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.670372    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.675111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.675006    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:44:27.675353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.675337    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:27.676277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.676256    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:44:27.676429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.676413    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:44:27.681666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.681639    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"]
Nov 02 17:44:27.771329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.771298    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.771509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.771482    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.771564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.771527    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79h9d\" (UniqueName: \"kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.873077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.873039    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.873274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.873085    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79h9d\" (UniqueName: \"kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.873342 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.873271    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.873944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.873914    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.876111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.876085    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.882883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.882820    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79h9d\" (UniqueName: \"kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d\") pod \"maas-api-revoke-sub-823c0021-rt9pg\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.996270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.996238    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:27.999832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.999618    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Nov 02 17:44:27.999832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:27.999678    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Nov 02 17:44:28.284622 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:28.284592    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"]
Nov 02 17:44:28.285162 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:44:28.285138    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c40dfa_984e_4eed_b3d6_4399a6b29887.slice/crio-0715ffffa16e5322c6f3eb2c474353d12b0dd991ace90c26113d2cdf06ad70bc WatchSource:0}: Error finding container 0715ffffa16e5322c6f3eb2c474353d12b0dd991ace90c26113d2cdf06ad70bc: Status 404 returned error can't find the container with id 0715ffffa16e5322c6f3eb2c474353d12b0dd991ace90c26113d2cdf06ad70bc
Nov 02 17:44:28.691511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:28.691475    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg" event={"ID":"83c40dfa-984e-4eed-b3d6-4399a6b29887","Type":"ContainerStarted","Data":"808314e5919f071f4f38b24741a39d9eb320425c364f9c8f9bf7e62578a62322"}
Nov 02 17:44:28.691694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:28.691579    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg" event={"ID":"83c40dfa-984e-4eed-b3d6-4399a6b29887","Type":"ContainerStarted","Data":"0715ffffa16e5322c6f3eb2c474353d12b0dd991ace90c26113d2cdf06ad70bc"}
Nov 02 17:44:28.710732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:28.710676    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg" podStartSLOduration=1.710663166 podStartE2EDuration="1.710663166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:28.70846926 +0000 UTC m=+681.287878647" watchObservedRunningTime="2026-11-02 17:44:28.710663166 +0000 UTC m=+681.290072546"
Nov 02 17:44:29.020881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:29.020818    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:29.021236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:29.021210    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:29.022254 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:29.022225    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 17:44:29.700255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:29.700220    5042 generic.go:356] "Generic (PLEG): container finished" podID="83c40dfa-984e-4eed-b3d6-4399a6b29887" containerID="808314e5919f071f4f38b24741a39d9eb320425c364f9c8f9bf7e62578a62322" exitCode=0
Nov 02 17:44:29.700714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:29.700318    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg" event={"ID":"83c40dfa-984e-4eed-b3d6-4399a6b29887","Type":"ContainerDied","Data":"808314e5919f071f4f38b24741a39d9eb320425c364f9c8f9bf7e62578a62322"}
Nov 02 17:44:30.545491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:30.545458    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:30.558473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:30.558446    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:30.719337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:30.719299    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:30.908983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:30.908956    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:31.025463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.025259    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d\" (UniqueName: \"kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d\") pod \"83c40dfa-984e-4eed-b3d6-4399a6b29887\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") "
Nov 02 17:44:31.025463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.025368    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca\") pod \"83c40dfa-984e-4eed-b3d6-4399a6b29887\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") "
Nov 02 17:44:31.025463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.025465    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca\") pod \"83c40dfa-984e-4eed-b3d6-4399a6b29887\" (UID: \"83c40dfa-984e-4eed-b3d6-4399a6b29887\") "
Nov 02 17:44:31.028232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.028195    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca" pod "83c40dfa-984e-4eed-b3d6-4399a6b29887" (UID: "83c40dfa-984e-4eed-b3d6-4399a6b29887"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:31.031000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.029174    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d" pod "83c40dfa-984e-4eed-b3d6-4399a6b29887" (UID: "83c40dfa-984e-4eed-b3d6-4399a6b29887"). InnerVolumeSpecName "kube-api-access-79h9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:31.035630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.035577    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca" pod "83c40dfa-984e-4eed-b3d6-4399a6b29887" (UID: "83c40dfa-984e-4eed-b3d6-4399a6b29887"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:31.127267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.127181    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:31.127267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.127213    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83c40dfa-984e-4eed-b3d6-4399a6b29887-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:31.127267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.127224    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79h9d\" (UniqueName: \"kubernetes.io/projected/83c40dfa-984e-4eed-b3d6-4399a6b29887-kube-api-access-79h9d\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:31.716533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.716494    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg" event={"ID":"83c40dfa-984e-4eed-b3d6-4399a6b29887","Type":"ContainerDied","Data":"0715ffffa16e5322c6f3eb2c474353d12b0dd991ace90c26113d2cdf06ad70bc"}
Nov 02 17:44:31.716533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.716524    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"
Nov 02 17:44:31.716802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:31.716558    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0715ffffa16e5322c6f3eb2c474353d12b0dd991ace90c26113d2cdf06ad70bc"
Nov 02 17:44:35.021876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.020927    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:35.206986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.206938    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:35.391607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.391511    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"]
Nov 02 17:44:35.392667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.392639    5042 cpu_manager.go:414] "containerMap: removing container" podUID="83c40dfa-984e-4eed-b3d6-4399a6b29887" containerName="revoke-keys"
Nov 02 17:44:35.392858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.392823    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c40dfa-984e-4eed-b3d6-4399a6b29887" containerName="revoke-keys"
Nov 02 17:44:35.393183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.393165    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c40dfa-984e-4eed-b3d6-4399a6b29887" containerName="revoke-keys"
Nov 02 17:44:35.425269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.424341    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"]
Nov 02 17:44:35.425269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.424476    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.437292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.437254    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.437577 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.437553    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.437728 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.437709    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.449147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.449014    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.449380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.449360    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5xg\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.449524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.449508    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.449800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.449782    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.449986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.449968    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.450129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.450115    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.575863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.572540    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.580932    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581018    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581048    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581094    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5xg\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581124    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581252    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581310    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.581366    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.583792    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.584139    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.584372    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.584598    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.648924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.643697    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.684865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.684147    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.817256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.817221    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:35.859914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.855916    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:35.890770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.887956    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:35.953777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.953732    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:44:36.013865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:35.979938    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:36.043241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.043195    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:36.044183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.044046    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5xg\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:36.112789 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.112756    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:44:36.164880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.141859    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:36.305926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.301981    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:44:36.355467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.355293    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:36.356022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.355999    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:44:36.823667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.823623    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:44:36.866625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.866524    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:36.867321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.867293    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:44:36.907739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.907702    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"]
Nov 02 17:44:36.924491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.923967    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.939867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.936456    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:36.939867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.937240    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945497    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945572    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945602    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945654    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945692    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945816    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945874    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945923    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j4lk\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:36.965900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:36.945964    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.025145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.024029    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:37.025145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.024661    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.053676    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.053733    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.053784    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j4lk\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.053827    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.053942    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.053992    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.054024    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.054079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.054119    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.055162    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.056098    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.057147    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.058309    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.079286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.058559    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.080505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.080058    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.123923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.097972    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:44:37.365867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.283899    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.365867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.305350    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"]
Nov 02 17:44:37.365867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.309536    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.365867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.336945    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:37.365867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.337662    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:44:37.418918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.373606    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:37.492925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.488525    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.570874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.555717    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j4lk\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:37.651878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.647748    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:38.006193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.882147    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:44:38.006193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.882201    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:44:38.006193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.884639    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:44:38.006193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.900985    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:38.006193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:37.939177    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:38.098385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.096923    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"]
Nov 02 17:44:38.237983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.236798    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:44:38.300250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.298574    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:38.300250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.298694    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:38.300250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.298751    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:38.339721 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.337218    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:44:38.438932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.436994    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:38.486085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.486043    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"]
Nov 02 17:44:38.549080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.547914    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"]
Nov 02 17:44:38.698926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.692377    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:38.710152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.710109    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" event={"ID":"fcba8ae9-d213-4e57-b57c-858c149f6038","Type":"ContainerStarted","Data":"ac08408e0c94e4a221572ba97d4a612d175a1a8f4f2988ef848d139838f2e03a"}
Nov 02 17:44:38.710421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.710399    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"]
Nov 02 17:44:38.721500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.721466    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:38.733194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.733164    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:44:38.736721 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.736690    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:44:38.737197 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.737174    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:44:38.737362 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.737347    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:38.737627 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.737605    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:44:38.893331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.893284    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:44:38.972862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:38.971623    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:44:39.010511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.010391    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxft\" (UniqueName: \"kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.011025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.010955    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.011486 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.011412    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.121875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.119879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxft\" (UniqueName: \"kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.121875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.119981    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.121875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.120111    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.121875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.121149    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.130909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.126117    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:39.130909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.129647    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.134815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.134757    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxft\" (UniqueName: \"kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft\") pod \"maas-api-revoke-sub-a140a353-89qv5\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.136112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.136043    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"]
Nov 02 17:44:39.142249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.142202    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:39.142490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.142459    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:39.142650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.142633    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:39.431146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.431074    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:39.678467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.678429    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" event={"ID":"04dbd50e-774a-48ff-9f8b-1337393e01a0","Type":"ContainerStarted","Data":"f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef"}
Nov 02 17:44:39.678647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.678495    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" event={"ID":"04dbd50e-774a-48ff-9f8b-1337393e01a0","Type":"ContainerStarted","Data":"443637b023d197e5a14a3b80221d0d4fc3c26b48280da9b4764c7f2bf5a4ce59"}
Nov 02 17:44:39.680760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.680731    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" event={"ID":"fcba8ae9-d213-4e57-b57c-858c149f6038","Type":"ContainerStarted","Data":"cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec"}
Nov 02 17:44:39.713368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.713312    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podStartSLOduration=3.7132973160000002 podStartE2EDuration="3.713297316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:39.71043908 +0000 UTC m=+692.289848467" watchObservedRunningTime="2026-11-02 17:44:39.713297316 +0000 UTC m=+692.292706701"
Nov 02 17:44:39.740336 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.740284    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podStartSLOduration=4.740268413 podStartE2EDuration="4.740268413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:39.736429513 +0000 UTC m=+692.315838902" watchObservedRunningTime="2026-11-02 17:44:39.740268413 +0000 UTC m=+692.319677835"
Nov 02 17:44:39.744883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:39.744855    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"]
Nov 02 17:44:39.745453 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:44:39.745427    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d3d523_9fb1_443c_9ebb_aa3f592c20bf.slice/crio-ba629fac8d54790178796955bd94e37fc9bcc164ae1a2cbfa7bebca059265c3e WatchSource:0}: Error finding container ba629fac8d54790178796955bd94e37fc9bcc164ae1a2cbfa7bebca059265c3e: Status 404 returned error can't find the container with id ba629fac8d54790178796955bd94e37fc9bcc164ae1a2cbfa7bebca059265c3e
Nov 02 17:44:40.153922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.151630    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:40.169882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.167128    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:40.169882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.167197    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:40.652918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.652881    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:40.654085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.654044    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:40.654215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.654113    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:40.714072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.714037    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5" event={"ID":"80d3d523-9fb1-443c-9ebb-aa3f592c20bf","Type":"ContainerStarted","Data":"9c6d6ed5575a89587b80dedceca80d1f4c34eba6d2d3adf96526bc87baabdaef"}
Nov 02 17:44:40.714298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.714281    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5" event={"ID":"80d3d523-9fb1-443c-9ebb-aa3f592c20bf","Type":"ContainerStarted","Data":"ba629fac8d54790178796955bd94e37fc9bcc164ae1a2cbfa7bebca059265c3e"}
Nov 02 17:44:40.748598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:40.748520    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5" podStartSLOduration=2.748500252 podStartE2EDuration="2.748500252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:40.748132565 +0000 UTC m=+693.327541951" watchObservedRunningTime="2026-11-02 17:44:40.748500252 +0000 UTC m=+693.327909823"
Nov 02 17:44:41.143302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:41.143266    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:41.143490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:41.143331    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:41.653703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:41.653660    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:41.654198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:41.653721    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:41.746615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:41.746577    5042 generic.go:356] "Generic (PLEG): container finished" podID="80d3d523-9fb1-443c-9ebb-aa3f592c20bf" containerID="9c6d6ed5575a89587b80dedceca80d1f4c34eba6d2d3adf96526bc87baabdaef" exitCode=0
Nov 02 17:44:41.746820 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:41.746680    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5" event={"ID":"80d3d523-9fb1-443c-9ebb-aa3f592c20bf","Type":"ContainerDied","Data":"9c6d6ed5575a89587b80dedceca80d1f4c34eba6d2d3adf96526bc87baabdaef"}
Nov 02 17:44:42.145974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:42.145220    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:42.145974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:42.145280    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:42.653733 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:42.653701    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:42.654170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:42.653758    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:43.155603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.155569    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:43.155901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.155872    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:43.268108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.268083    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:43.356388 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.356354    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft\" (UniqueName: \"kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft\") pod \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") "
Nov 02 17:44:43.357058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.357037    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca\") pod \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") "
Nov 02 17:44:43.357371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.357313    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca\") pod \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\" (UID: \"80d3d523-9fb1-443c-9ebb-aa3f592c20bf\") "
Nov 02 17:44:43.357517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.357469    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca" pod "80d3d523-9fb1-443c-9ebb-aa3f592c20bf" (UID: "80d3d523-9fb1-443c-9ebb-aa3f592c20bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:43.357993 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.357969    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:43.359967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.359932    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca" pod "80d3d523-9fb1-443c-9ebb-aa3f592c20bf" (UID: "80d3d523-9fb1-443c-9ebb-aa3f592c20bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:43.360061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.360004    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft" pod "80d3d523-9fb1-443c-9ebb-aa3f592c20bf" (UID: "80d3d523-9fb1-443c-9ebb-aa3f592c20bf"). InnerVolumeSpecName "kube-api-access-tzxft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:43.459544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.459495    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxft\" (UniqueName: \"kubernetes.io/projected/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-kube-api-access-tzxft\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:43.459544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.459540    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80d3d523-9fb1-443c-9ebb-aa3f592c20bf-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:43.498929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.497916    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g" podUID="98652732-484b-4eeb-8292-9a0d4be2f713" containerName="payload-processing" containerID="cri-o://d275f576f48125a31057b6d4394a392b3108313ae6212a15056700e0e973ea40" gracePeriod=25
Nov 02 17:44:43.653822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.653731    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:43.653822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.653797    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:43.851937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.851384    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5" event={"ID":"80d3d523-9fb1-443c-9ebb-aa3f592c20bf","Type":"ContainerDied","Data":"ba629fac8d54790178796955bd94e37fc9bcc164ae1a2cbfa7bebca059265c3e"}
Nov 02 17:44:43.851937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.851452    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba629fac8d54790178796955bd94e37fc9bcc164ae1a2cbfa7bebca059265c3e"
Nov 02 17:44:43.851937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.851543    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"
Nov 02 17:44:43.940111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.916480    5042 generic.go:356] "Generic (PLEG): container finished" podID="98652732-484b-4eeb-8292-9a0d4be2f713" containerID="d275f576f48125a31057b6d4394a392b3108313ae6212a15056700e0e973ea40" exitCode=0
Nov 02 17:44:43.940111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:43.916545    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g" event={"ID":"98652732-484b-4eeb-8292-9a0d4be2f713","Type":"ContainerDied","Data":"d275f576f48125a31057b6d4394a392b3108313ae6212a15056700e0e973ea40"}
Nov 02 17:44:44.158922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.153285    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:44.158922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.153346    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:44.568309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.568277    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:44:44.713182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.711969    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:44.713182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.712029    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:44.817866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.797003    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume\") pod \"98652732-484b-4eeb-8292-9a0d4be2f713\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") "
Nov 02 17:44:44.817866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.797061    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials\") pod \"98652732-484b-4eeb-8292-9a0d4be2f713\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") "
Nov 02 17:44:44.817866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.797098    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume\") pod \"98652732-484b-4eeb-8292-9a0d4be2f713\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") "
Nov 02 17:44:44.817866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.797132    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb\") pod \"98652732-484b-4eeb-8292-9a0d4be2f713\" (UID: \"98652732-484b-4eeb-8292-9a0d4be2f713\") "
Nov 02 17:44:44.817866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.801619    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume" pod "98652732-484b-4eeb-8292-9a0d4be2f713" (UID: "98652732-484b-4eeb-8292-9a0d4be2f713"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:44.817866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.802002    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume" pod "98652732-484b-4eeb-8292-9a0d4be2f713" (UID: "98652732-484b-4eeb-8292-9a0d4be2f713"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:44.857701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.857660    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb" pod "98652732-484b-4eeb-8292-9a0d4be2f713" (UID: "98652732-484b-4eeb-8292-9a0d4be2f713"). InnerVolumeSpecName "kube-api-access-gjkdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:44.859471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.859429    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials" pod "98652732-484b-4eeb-8292-9a0d4be2f713" (UID: "98652732-484b-4eeb-8292-9a0d4be2f713"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:44.908862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.907481    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:44.908862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.907517    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:44.908862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.907532    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/98652732-484b-4eeb-8292-9a0d4be2f713-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:44.908862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.907548    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjkdb\" (UniqueName: \"kubernetes.io/projected/98652732-484b-4eeb-8292-9a0d4be2f713-kube-api-access-gjkdb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:44.947132 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:44.947087    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:45.029781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.026281    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g" event={"ID":"98652732-484b-4eeb-8292-9a0d4be2f713","Type":"ContainerDied","Data":"d97096b58c954888f411afc390794571bafee7d9d790fcab784559351bff8197"}
Nov 02 17:44:45.029781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.026356    5042 scope.go:122] "RemoveContainer" containerID="d275f576f48125a31057b6d4394a392b3108313ae6212a15056700e0e973ea40"
Nov 02 17:44:45.029781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.026535    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"
Nov 02 17:44:45.080331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.079932    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:45.151873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.150145    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:45.151873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.150204    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:45.187897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.181071    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"]
Nov 02 17:44:45.224866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.220873    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bb6976b77-7rk7g"]
Nov 02 17:44:45.671386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.669897    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:45.671386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.669975    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:45.671386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.670622    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98652732-484b-4eeb-8292-9a0d4be2f713" path="/var/lib/kubelet/pods/98652732-484b-4eeb-8292-9a0d4be2f713/volumes"
Nov 02 17:44:45.739194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.739125    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:44:45.780765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.776939    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:44:45.953873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.949997    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:45.971918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.962950    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:45.973141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:45.973114    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:44:46.219926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.219253    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:46.219926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.219316    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:46.241868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.229760    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:46.241868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.230192    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:44:46.241868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.231659    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:46.241868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.236520    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:46.618990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.618895    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"]
Nov 02 17:44:46.620003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.619977    5042 cpu_manager.go:414] "containerMap: removing container" podUID="98652732-484b-4eeb-8292-9a0d4be2f713" containerName="payload-processing"
Nov 02 17:44:46.620145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.620133    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98652732-484b-4eeb-8292-9a0d4be2f713" containerName="payload-processing"
Nov 02 17:44:46.620250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.620239    5042 cpu_manager.go:414] "containerMap: removing container" podUID="80d3d523-9fb1-443c-9ebb-aa3f592c20bf" containerName="revoke-keys"
Nov 02 17:44:46.620334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.620324    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d3d523-9fb1-443c-9ebb-aa3f592c20bf" containerName="revoke-keys"
Nov 02 17:44:46.620627 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.620614    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="98652732-484b-4eeb-8292-9a0d4be2f713" containerName="payload-processing"
Nov 02 17:44:46.620718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.620708    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d3d523-9fb1-443c-9ebb-aa3f592c20bf" containerName="revoke-keys"
Nov 02 17:44:46.674873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.672255    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:46.674873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.672315    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Nov 02 17:44:46.674873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.672866    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"]
Nov 02 17:44:46.674873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.672972    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.697623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.697595    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 17:44:46.711918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.697983    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 17:44:46.712261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.712239    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 17:44:46.712665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.712643    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-phftv\""
Nov 02 17:44:46.785257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.772581    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:46.785257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.773352    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:44:46.829560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.800317    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:46.829560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.801035    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.835784    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.835854    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.835892    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.835929    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.835956    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.835982    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gwb\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.856495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.836070    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.963867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.938756    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:46.989867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.964199    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.024871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.991039    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.024871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.992593    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:44:47.024871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.992681    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:47.024871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:46.992716    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.065492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.065441    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.065797 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.065758    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.066012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.065983    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.066137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.066122    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.066281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.066266    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gwb\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.066507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.066491    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.098478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.098445    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.178125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.153608    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:47.178443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.178403    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:47.245930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.240311    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.245930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.240431    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.250872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.246268    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:47.261905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.251540    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:47.309864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.256309    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:47.319019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.318985    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:44:47.377066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.377012    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.429971 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.429918    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.445129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.444350    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.457112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.457028    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:44:47.674946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.661992    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:47.788264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.788215    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.933098 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.910165    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.935992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.819790    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.972666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:47.972630    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:44:48.048878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.016793    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.285930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.193322    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:48.285930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.282821    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:48.285930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.258046    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.285930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.246949    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:44:48.353729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.353693    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gwb\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bktgf\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:48.379069 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.379030    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:48.379363 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.379332    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.462096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.460910    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:44:48.508879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.501953    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.533394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.533034    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:48.533394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.533102    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.618425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:48.618091    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:49.055867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.053579    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:49.055867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.054094    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.087867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.086494    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:49.087867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.086591    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:49.087867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.087489    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:49.087867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.087576    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.144070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.144036    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:49.144376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.144333    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Nov 02 17:44:49.157907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.157879    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:44:49.182160 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:44:49.182126    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:44:49.287072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.287043    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:44:49.509040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.508107    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:49.559865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:49.558054    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.096944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.071900    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.142321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.122106    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:44:50.142321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.098089    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.179351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.179313    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:50.179613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.179587    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.214639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.213980    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:44:50.214639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.214026    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"]
Nov 02 17:44:50.214639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.214082    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:44:50.214639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.214113    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:44:50.215750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.215722    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:44:50.218267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.218246    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:44:50.528093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.527992    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:50.530298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.530275    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:50.533221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.533201    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 17:44:50.569806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.569768    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" event={"ID":"78d07225-594c-4799-b7ca-6c7290cc9202","Type":"ContainerStarted","Data":"ce8a00f6330472f4a4649dd57ce463ad7f6d0d10dd1bd488068fd9ebac15865c"}
Nov 02 17:44:50.818727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.818653    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:50.819955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.819931    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:50.820579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.820556    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 17:44:50.993309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:50.992879    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"]
Nov 02 17:44:51.007226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.007195    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"]
Nov 02 17:44:51.007426 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.007292    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.012636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.012149    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:44:51.012900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.012820    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:44:51.012983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.012955    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:44:51.013039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.013012    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:51.090306 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.090209    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.090476 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.090350    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.090562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.090540    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482kn\" (UniqueName: \"kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.191750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.191710    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.192200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.191925    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482kn\" (UniqueName: \"kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.192200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.191990    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.192606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.192581    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.194697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.194667    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.201929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.201900    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482kn\" (UniqueName: \"kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn\") pod \"maas-api-revoke-sub-aa0d6de8-dfd5x\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.330035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.329989    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:51.634078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:51.634043    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"]
Nov 02 17:44:52.595176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:52.595139    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x" event={"ID":"78e1fbbd-28ff-4fe3-bbd3-871b37665b01","Type":"ContainerStarted","Data":"e51f50c2fd9e9002480059193ab20017dc6ac1f2d7d091d44b02857bc0de1148"}
Nov 02 17:44:52.595666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:52.595200    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x" event={"ID":"78e1fbbd-28ff-4fe3-bbd3-871b37665b01","Type":"ContainerStarted","Data":"284c13d4c1035f6d5182765f8db57da97c0837d31d9181bfe7e20f8f0852e387"}
Nov 02 17:44:52.617939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:52.617870    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x" podStartSLOduration=2.617854738 podStartE2EDuration="2.617854738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:52.613451313 +0000 UTC m=+705.192860704" watchObservedRunningTime="2026-11-02 17:44:52.617854738 +0000 UTC m=+705.197264121"
Nov 02 17:44:53.360611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.360574    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:53.360758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.360651    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:53.610189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.610093    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" event={"ID":"78d07225-594c-4799-b7ca-6c7290cc9202","Type":"ContainerStarted","Data":"5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1"}
Nov 02 17:44:53.611234 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.611210    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:44:53.613314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.613287    5042 generic.go:356] "Generic (PLEG): container finished" podID="78e1fbbd-28ff-4fe3-bbd3-871b37665b01" containerID="e51f50c2fd9e9002480059193ab20017dc6ac1f2d7d091d44b02857bc0de1148" exitCode=0
Nov 02 17:44:53.613426 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.613331    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x" event={"ID":"78e1fbbd-28ff-4fe3-bbd3-871b37665b01","Type":"ContainerDied","Data":"e51f50c2fd9e9002480059193ab20017dc6ac1f2d7d091d44b02857bc0de1148"}
Nov 02 17:44:53.641583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:53.639377    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" podStartSLOduration=4.412318305 podStartE2EDuration="7.639358817s" totalImagesPullingTime="3.227040512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:53.633645046 +0000 UTC m=+706.213054433" watchObservedRunningTime="2026-11-02 17:44:53.639358817 +0000 UTC m=+706.218768205"
Nov 02 17:44:54.653466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.653427    5042 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bktgf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:44:54.653926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.653496    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" podUID="78d07225-594c-4799-b7ca-6c7290cc9202" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:44:54.916493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.916412    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:54.968122 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.968084    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca\") pod \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") "
Nov 02 17:44:54.968286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.968173    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca\") pod \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") "
Nov 02 17:44:54.968362 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.968290    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn\" (UniqueName: \"kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn\") pod \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\" (UID: \"78e1fbbd-28ff-4fe3-bbd3-871b37665b01\") "
Nov 02 17:44:54.968546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.968518    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca" pod "78e1fbbd-28ff-4fe3-bbd3-871b37665b01" (UID: "78e1fbbd-28ff-4fe3-bbd3-871b37665b01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:54.968992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.968967    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:54.970607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.970569    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca" pod "78e1fbbd-28ff-4fe3-bbd3-871b37665b01" (UID: "78e1fbbd-28ff-4fe3-bbd3-871b37665b01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:54.970869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:54.970826    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn" pod "78e1fbbd-28ff-4fe3-bbd3-871b37665b01" (UID: "78e1fbbd-28ff-4fe3-bbd3-871b37665b01"). InnerVolumeSpecName "kube-api-access-482kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:55.070141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:55.070102    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:55.070141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:55.070128    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482kn\" (UniqueName: \"kubernetes.io/projected/78e1fbbd-28ff-4fe3-bbd3-871b37665b01-kube-api-access-482kn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:44:55.664884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:55.664852    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"
Nov 02 17:44:55.665430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:55.664881    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x" event={"ID":"78e1fbbd-28ff-4fe3-bbd3-871b37665b01","Type":"ContainerDied","Data":"284c13d4c1035f6d5182765f8db57da97c0837d31d9181bfe7e20f8f0852e387"}
Nov 02 17:44:55.665430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:55.664946    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="284c13d4c1035f6d5182765f8db57da97c0837d31d9181bfe7e20f8f0852e387"
Nov 02 17:44:55.666183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:44:55.666161    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 17:45:00.170908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.170879    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"]
Nov 02 17:45:00.171419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.171405    5042 cpu_manager.go:414] "containerMap: removing container" podUID="78e1fbbd-28ff-4fe3-bbd3-871b37665b01" containerName="revoke-keys"
Nov 02 17:45:00.171463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.171420    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e1fbbd-28ff-4fe3-bbd3-871b37665b01" containerName="revoke-keys"
Nov 02 17:45:00.171531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.171522    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e1fbbd-28ff-4fe3-bbd3-871b37665b01" containerName="revoke-keys"
Nov 02 17:45:00.176897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.176870    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.178641 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.178618    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"]
Nov 02 17:45:00.184171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.184151    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.184745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.184328    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"]
Nov 02 17:45:00.189776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.189757    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.220447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.220420    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"]
Nov 02 17:45:00.223123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.223092    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:45:00.223282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.223189    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:45:00.223282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.223230    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:45:00.229858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.229811    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"]
Nov 02 17:45:00.244464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.244432    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.244628 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.244579    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.244688 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.244665    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.244741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.244713    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pndw\" (UniqueName: \"kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.244741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.244733    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tn76\" (UniqueName: \"kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.244907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.244763    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvrq\" (UniqueName: \"kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.245990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.245961    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"]
Nov 02 17:45:00.251800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.249622    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"]
Nov 02 17:45:00.255687 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.255646    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"]
Nov 02 17:45:00.255806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.255754    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.346154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346122    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.346346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346184    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-xqq58\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.346346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346205    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.346346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346244    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pndw\" (UniqueName: \"kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.346346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346265    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tn76\" (UniqueName: \"kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.346578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346335    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvrq\" (UniqueName: \"kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.346578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346466    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b94x\" (UniqueName: \"kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x\") pod \"maas-api-key-cleanup-29894025-xqq58\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.346578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346507    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.346928 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346906    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.347005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.346925    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.347153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.347136    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.360515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.360483    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvrq\" (UniqueName: \"kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.360938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.360921    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pndw\" (UniqueName: \"kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.361707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.361681    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tn76\" (UniqueName: \"kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.447644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.447611    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b94x\" (UniqueName: \"kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x\") pod \"maas-api-key-cleanup-29894025-xqq58\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.447878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.447734    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-xqq58\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.448400 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.448373    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-xqq58\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.457140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.457114    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b94x\" (UniqueName: \"kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x\") pod \"maas-api-key-cleanup-29894025-xqq58\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.533111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.533062    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:00.564957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.564923    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:00.616004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.613080    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:00.636062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.636020    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:00.844963 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:00.844043    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669d4462_23c5_4b90_b3c0_4175661fbec3.slice/crio-c25cac06c3fbbf9a8843303463dd5b75d5f58a6cd89e07e20f2b406aa9988e14 WatchSource:0}: Error finding container c25cac06c3fbbf9a8843303463dd5b75d5f58a6cd89e07e20f2b406aa9988e14: Status 404 returned error can't find the container with id c25cac06c3fbbf9a8843303463dd5b75d5f58a6cd89e07e20f2b406aa9988e14
Nov 02 17:45:00.844963 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.844703    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"]
Nov 02 17:45:00.889412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.889379    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"]
Nov 02 17:45:00.951207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.951176    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"]
Nov 02 17:45:00.952273 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:00.952242    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7780aee0_96a8_4ed3_bb27_cab9dffe3a04.slice/crio-81eb28dc88b8af2c5904c923ce410ded4e7749895b24709c2b9f7eab3bd53c9c WatchSource:0}: Error finding container 81eb28dc88b8af2c5904c923ce410ded4e7749895b24709c2b9f7eab3bd53c9c: Status 404 returned error can't find the container with id 81eb28dc88b8af2c5904c923ce410ded4e7749895b24709c2b9f7eab3bd53c9c
Nov 02 17:45:00.998904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:00.998871    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"]
Nov 02 17:45:01.735728 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.735680    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerStarted","Data":"7ac9e91a3c92fec13539e916eb5864a70f350b151e7d73ebc473456037207193"}
Nov 02 17:45:01.736251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.735747    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerStarted","Data":"81eb28dc88b8af2c5904c923ce410ded4e7749895b24709c2b9f7eab3bd53c9c"}
Nov 02 17:45:01.737668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.737636    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerStarted","Data":"e04870fadbcd33dd92d95776916adc48dd8fbbc2901c95c1526f81c9351555e5"}
Nov 02 17:45:01.737811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.737694    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerStarted","Data":"c25cac06c3fbbf9a8843303463dd5b75d5f58a6cd89e07e20f2b406aa9988e14"}
Nov 02 17:45:01.739969 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.739907    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerStarted","Data":"f3b9aa619ac4104c64675017d6ab37b4b70a9da97f0d61d8d0a28e242f9c7721"}
Nov 02 17:45:01.740081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.739998    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerStarted","Data":"8a94e3a7c4054d48192787ea1c2a5692e3003d920da42489f2967cc0529ff8b5"}
Nov 02 17:45:01.741825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.741797    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerStarted","Data":"42c86fd0ebafb030bd6f50ff47f3f6f2fca68066f9e7b2706198e25b3b7f1dbf"}
Nov 02 17:45:01.741969 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.741879    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerStarted","Data":"2b2afcbd83fd9d8d137c80b93e30bfe1c31c4dcc71826d81865081807269d392"}
Nov 02 17:45:01.757519 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.757467    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" podStartSLOduration=1.75745337 podStartE2EDuration="1.75745337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.753215107 +0000 UTC m=+714.332624494" watchObservedRunningTime="2026-11-02 17:45:01.75745337 +0000 UTC m=+714.336862756"
Nov 02 17:45:01.778389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.778318    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" podStartSLOduration=1.7782942959999999 podStartE2EDuration="1.778294296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.771363915 +0000 UTC m=+714.350773303" watchObservedRunningTime="2026-11-02 17:45:01.778294296 +0000 UTC m=+714.357703683"
Nov 02 17:45:01.810179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.810100    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" podStartSLOduration=1.810079555 podStartE2EDuration="1.810079555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.808331918 +0000 UTC m=+714.387741315" watchObservedRunningTime="2026-11-02 17:45:01.810079555 +0000 UTC m=+714.389488944"
Nov 02 17:45:01.844918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:01.835526    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" podStartSLOduration=1.835500495 podStartE2EDuration="1.835500495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.827429267 +0000 UTC m=+714.406838655" watchObservedRunningTime="2026-11-02 17:45:01.835500495 +0000 UTC m=+714.414909881"
Nov 02 17:45:02.753894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.753809    5042 generic.go:356] "Generic (PLEG): container finished" podID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerID="7ac9e91a3c92fec13539e916eb5864a70f350b151e7d73ebc473456037207193" exitCode=60
Nov 02 17:45:02.754397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.753972    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerDied","Data":"7ac9e91a3c92fec13539e916eb5864a70f350b151e7d73ebc473456037207193"}
Nov 02 17:45:02.754755 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.754473    5042 scope.go:122] "RemoveContainer" containerID="7ac9e91a3c92fec13539e916eb5864a70f350b151e7d73ebc473456037207193"
Nov 02 17:45:02.756493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.756470    5042 generic.go:356] "Generic (PLEG): container finished" podID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerID="e04870fadbcd33dd92d95776916adc48dd8fbbc2901c95c1526f81c9351555e5" exitCode=60
Nov 02 17:45:02.756714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.756554    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerDied","Data":"e04870fadbcd33dd92d95776916adc48dd8fbbc2901c95c1526f81c9351555e5"}
Nov 02 17:45:02.756811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.756743    5042 scope.go:122] "RemoveContainer" containerID="e04870fadbcd33dd92d95776916adc48dd8fbbc2901c95c1526f81c9351555e5"
Nov 02 17:45:02.763930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.763907    5042 generic.go:356] "Generic (PLEG): container finished" podID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerID="f3b9aa619ac4104c64675017d6ab37b4b70a9da97f0d61d8d0a28e242f9c7721" exitCode=60
Nov 02 17:45:02.764023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.763983    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerDied","Data":"f3b9aa619ac4104c64675017d6ab37b4b70a9da97f0d61d8d0a28e242f9c7721"}
Nov 02 17:45:02.764268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.764251    5042 scope.go:122] "RemoveContainer" containerID="f3b9aa619ac4104c64675017d6ab37b4b70a9da97f0d61d8d0a28e242f9c7721"
Nov 02 17:45:02.767394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.767374    5042 generic.go:356] "Generic (PLEG): container finished" podID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerID="42c86fd0ebafb030bd6f50ff47f3f6f2fca68066f9e7b2706198e25b3b7f1dbf" exitCode=60
Nov 02 17:45:02.767636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.767621    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerDied","Data":"42c86fd0ebafb030bd6f50ff47f3f6f2fca68066f9e7b2706198e25b3b7f1dbf"}
Nov 02 17:45:02.768061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:02.768044    5042 scope.go:122] "RemoveContainer" containerID="42c86fd0ebafb030bd6f50ff47f3f6f2fca68066f9e7b2706198e25b3b7f1dbf"
Nov 02 17:45:03.845946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:03.839946    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerStarted","Data":"719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f"}
Nov 02 17:45:03.864409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:03.864372    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerStarted","Data":"d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2"}
Nov 02 17:45:03.868038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:03.867973    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerStarted","Data":"3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed"}
Nov 02 17:45:04.890877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.890806    5042 generic.go:356] "Generic (PLEG): container finished" podID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerID="3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed" exitCode=60
Nov 02 17:45:04.891404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.890908    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerDied","Data":"3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed"}
Nov 02 17:45:04.891404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.891013    5042 scope.go:122] "RemoveContainer" containerID="7ac9e91a3c92fec13539e916eb5864a70f350b151e7d73ebc473456037207193"
Nov 02 17:45:04.891404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.891206    5042 scope.go:122] "RemoveContainer" containerID="3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed"
Nov 02 17:45:04.891621 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:04.891518    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv_odh-ai-gateway-infra(7780aee0-96a8-4ed3-bb27-cab9dffe3a04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04"
Nov 02 17:45:04.893424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.893391    5042 generic.go:356] "Generic (PLEG): container finished" podID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerID="4b5b2acea64acebc6fd0c66b310da6fbe6f59dedc37b53c4232df11ca5cc75f0" exitCode=60
Nov 02 17:45:04.893535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.893467    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerDied","Data":"4b5b2acea64acebc6fd0c66b310da6fbe6f59dedc37b53c4232df11ca5cc75f0"}
Nov 02 17:45:04.893826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.893808    5042 scope.go:122] "RemoveContainer" containerID="4b5b2acea64acebc6fd0c66b310da6fbe6f59dedc37b53c4232df11ca5cc75f0"
Nov 02 17:45:04.894150 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:04.894122    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd_odh-ai-gateway-infra(669d4462-23c5-4b90-b3c0-4175661fbec3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3"
Nov 02 17:45:04.896036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.896012    5042 generic.go:356] "Generic (PLEG): container finished" podID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerID="719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f" exitCode=60
Nov 02 17:45:04.896117 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.896087    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerDied","Data":"719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f"}
Nov 02 17:45:04.896316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.896300    5042 scope.go:122] "RemoveContainer" containerID="719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f"
Nov 02 17:45:04.896759 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:04.896664    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-xqq58_odh-ai-gateway-infra(f529fc7a-652f-4b7c-8581-14537f0438e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0"
Nov 02 17:45:04.898473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.898453    5042 generic.go:356] "Generic (PLEG): container finished" podID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerID="d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2" exitCode=60
Nov 02 17:45:04.898586 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.898538    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerDied","Data":"d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2"}
Nov 02 17:45:04.898772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.898754    5042 scope.go:122] "RemoveContainer" containerID="d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2"
Nov 02 17:45:04.899037 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:04.899015    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj_odh-ai-gateway-infra(0360c96d-a7ea-4ac7-9424-26ae315830dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc"
Nov 02 17:45:04.907097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.907071    5042 scope.go:122] "RemoveContainer" containerID="e04870fadbcd33dd92d95776916adc48dd8fbbc2901c95c1526f81c9351555e5"
Nov 02 17:45:04.929890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.929863    5042 scope.go:122] "RemoveContainer" containerID="f3b9aa619ac4104c64675017d6ab37b4b70a9da97f0d61d8d0a28e242f9c7721"
Nov 02 17:45:04.958852 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:04.958807    5042 scope.go:122] "RemoveContainer" containerID="42c86fd0ebafb030bd6f50ff47f3f6f2fca68066f9e7b2706198e25b3b7f1dbf"
Nov 02 17:45:05.793368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.786390    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"]
Nov 02 17:45:05.817932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.815119    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.839599 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.839522    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"]
Nov 02 17:45:05.860919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.860888    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.861295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.861270    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.861491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.861469    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7446k\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.861667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.861651    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.861879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.861862    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.862065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.862048    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.862217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.862201    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.862372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.862357    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.862529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.862515    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.928862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.928271    5042 scope.go:122] "RemoveContainer" containerID="3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed"
Nov 02 17:45:05.928862 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:05.928624    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv_odh-ai-gateway-infra(7780aee0-96a8-4ed3-bb27-cab9dffe3a04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04"
Nov 02 17:45:05.954698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.934501    5042 scope.go:122] "RemoveContainer" containerID="719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f"
Nov 02 17:45:05.954698 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:05.934821    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-xqq58_odh-ai-gateway-infra(f529fc7a-652f-4b7c-8581-14537f0438e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0"
Nov 02 17:45:05.954698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.937459    5042 scope.go:122] "RemoveContainer" containerID="d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2"
Nov 02 17:45:05.954698 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:05.937736    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj_odh-ai-gateway-infra(0360c96d-a7ea-4ac7-9424-26ae315830dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc"
Nov 02 17:45:05.963584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963552    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.963771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963624    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.963771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963655    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.963771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963681    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.963771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963721    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963878    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.963984    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.964015    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.964230    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.964731    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7446k\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.964816    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.964961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.964867    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.965517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.965485    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.965706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.965688    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.967150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.967129    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.972800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.972777    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.976349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.976322    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:05.982034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:05.981995    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7446k\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k\") pod \"e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") " pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:06.170978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.170479    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:06.472788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.472755    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"]
Nov 02 17:45:06.474183 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:06.474157    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d62cc7_9fc5_455e_81a8_d71472720401.slice/crio-47cf1426476936e51b030469cabe426e1e65da6574a61f4edbc130bff4d3fc10 WatchSource:0}: Error finding container 47cf1426476936e51b030469cabe426e1e65da6574a61f4edbc130bff4d3fc10: Status 404 returned error can't find the container with id 47cf1426476936e51b030469cabe426e1e65da6574a61f4edbc130bff4d3fc10
Nov 02 17:45:06.477030 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.476997    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:06.477134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.477075    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:06.477134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.477107    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:06.878666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.878586    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"]
Nov 02 17:45:06.882858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.882818    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:06.893912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.893886    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:06.900295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.900263    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"]
Nov 02 17:45:06.952324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.952288    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" event={"ID":"68d62cc7-9fc5-455e-81a8-d71472720401","Type":"ContainerStarted","Data":"849dd99cd9a9c862fcff142ce5109216f838cebf3860dc443f989a81579c6839"}
Nov 02 17:45:06.952818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.952346    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" event={"ID":"68d62cc7-9fc5-455e-81a8-d71472720401","Type":"ContainerStarted","Data":"47cf1426476936e51b030469cabe426e1e65da6574a61f4edbc130bff4d3fc10"}
Nov 02 17:45:06.982474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.982425    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:06.982689 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.982526    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:06.982771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.982715    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444l5\" (UniqueName: \"kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:06.992682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:06.992456    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podStartSLOduration=1.992438021 podStartE2EDuration="1.992438021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:06.987002748 +0000 UTC m=+719.566412135" watchObservedRunningTime="2026-11-02 17:45:06.992438021 +0000 UTC m=+719.571847408"
Nov 02 17:45:07.085712 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.085669    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.086607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.086581    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.087736 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.087715    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-444l5\" (UniqueName: \"kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.088520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.087522    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.095002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.094978    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.111360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.111326    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-444l5\" (UniqueName: \"kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5\") pod \"maas-api-revoke-sub-1b2ac45e-stjsc\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.171113 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.171029    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:07.191866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.184008    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:07.191866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.184073    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:07.206865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.203481    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:07.530923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.530890    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"]
Nov 02 17:45:07.961208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.961171    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc" event={"ID":"6be480d4-5ae6-488b-b883-35455abb3e0b","Type":"ContainerStarted","Data":"de7fe871569b59fd8b61d672eebb650f6e65d93e4282c991baae1dd1805fa67f"}
Nov 02 17:45:07.961636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:07.961235    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc" event={"ID":"6be480d4-5ae6-488b-b883-35455abb3e0b","Type":"ContainerStarted","Data":"a486a86eb8f304dc5ce8678a349ac1408a7591feabab96e584155c2ef83ab0a3"}
Nov 02 17:45:08.008068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:08.002832    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc" podStartSLOduration=2.002807833 podStartE2EDuration="2.002807833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:07.982945679 +0000 UTC m=+720.562355065" watchObservedRunningTime="2026-11-02 17:45:08.002807833 +0000 UTC m=+720.582217220"
Nov 02 17:45:08.171744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:08.171659    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:08.171744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:08.171712    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:08.970750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:08.970713    5042 generic.go:356] "Generic (PLEG): container finished" podID="6be480d4-5ae6-488b-b883-35455abb3e0b" containerID="de7fe871569b59fd8b61d672eebb650f6e65d93e4282c991baae1dd1805fa67f" exitCode=0
Nov 02 17:45:08.971208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:08.970801    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc" event={"ID":"6be480d4-5ae6-488b-b883-35455abb3e0b","Type":"ContainerDied","Data":"de7fe871569b59fd8b61d672eebb650f6e65d93e4282c991baae1dd1805fa67f"}
Nov 02 17:45:09.171396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:09.171356    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:09.171595 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:09.171414    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:10.171545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.171505    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:10.172083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.171569    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:10.244601 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.244575    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:10.428997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.428913    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca\") pod \"6be480d4-5ae6-488b-b883-35455abb3e0b\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") "
Nov 02 17:45:10.429146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.429027    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5\" (UniqueName: \"kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5\") pod \"6be480d4-5ae6-488b-b883-35455abb3e0b\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") "
Nov 02 17:45:10.429187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.429146    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca\") pod \"6be480d4-5ae6-488b-b883-35455abb3e0b\" (UID: \"6be480d4-5ae6-488b-b883-35455abb3e0b\") "
Nov 02 17:45:10.429611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.429574    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca" pod "6be480d4-5ae6-488b-b883-35455abb3e0b" (UID: "6be480d4-5ae6-488b-b883-35455abb3e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:10.432589 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.432489    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5" pod "6be480d4-5ae6-488b-b883-35455abb3e0b" (UID: "6be480d4-5ae6-488b-b883-35455abb3e0b"). InnerVolumeSpecName "kube-api-access-444l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:10.432589 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.432550    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca" pod "6be480d4-5ae6-488b-b883-35455abb3e0b" (UID: "6be480d4-5ae6-488b-b883-35455abb3e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:10.531065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.531031    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:10.531065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.531065    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be480d4-5ae6-488b-b883-35455abb3e0b-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:10.531287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:10.531076    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444l5\" (UniqueName: \"kubernetes.io/projected/6be480d4-5ae6-488b-b883-35455abb3e0b-kube-api-access-444l5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:11.017499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:11.016246    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc" event={"ID":"6be480d4-5ae6-488b-b883-35455abb3e0b","Type":"ContainerDied","Data":"a486a86eb8f304dc5ce8678a349ac1408a7591feabab96e584155c2ef83ab0a3"}
Nov 02 17:45:11.017499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:11.016297    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a486a86eb8f304dc5ce8678a349ac1408a7591feabab96e584155c2ef83ab0a3"
Nov 02 17:45:11.017499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:11.016354    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"
Nov 02 17:45:11.171535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:11.171492    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:11.171701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:11.171565    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:12.175868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:12.173948    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:12.175868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:12.174012    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:13.178291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:13.178252    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 17:45:13.178960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:13.178926    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 17:45:14.710101 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:14.710066    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:14.711654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:14.710666    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:14.711654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:14.711552    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 17:45:15.540514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:15.540460    5042 scope.go:122] "RemoveContainer" containerID="719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f"
Nov 02 17:45:16.104852 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:16.104807    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerStarted","Data":"f50345578cc50ac7a5cb4608a68ed549f35948844bd77744e01bdd47e9e2652c"}
Nov 02 17:45:16.535729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:16.535694    5042 scope.go:122] "RemoveContainer" containerID="d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2"
Nov 02 17:45:17.120954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.120922    5042 generic.go:356] "Generic (PLEG): container finished" podID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerID="f50345578cc50ac7a5cb4608a68ed549f35948844bd77744e01bdd47e9e2652c" exitCode=60
Nov 02 17:45:17.121360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.121000    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerDied","Data":"f50345578cc50ac7a5cb4608a68ed549f35948844bd77744e01bdd47e9e2652c"}
Nov 02 17:45:17.121360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.121084    5042 scope.go:122] "RemoveContainer" containerID="719eb7ac2b03b625af1e0229cf91ee46d8c826a011631deb98c6febbb802fc2f"
Nov 02 17:45:17.121664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.121515    5042 scope.go:122] "RemoveContainer" containerID="f50345578cc50ac7a5cb4608a68ed549f35948844bd77744e01bdd47e9e2652c"
Nov 02 17:45:17.121932 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:17.121907    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-xqq58_odh-ai-gateway-infra(f529fc7a-652f-4b7c-8581-14537f0438e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0"
Nov 02 17:45:17.124720 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.124694    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerStarted","Data":"ae42f994324c7fd6a03f1012f15489fe325006deb381352238aaab85a5b0649c"}
Nov 02 17:45:17.140771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.140722    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"]
Nov 02 17:45:17.359544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.359492    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"]
Nov 02 17:45:17.360182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.360162    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6be480d4-5ae6-488b-b883-35455abb3e0b" containerName="revoke-keys"
Nov 02 17:45:17.360182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.360183    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be480d4-5ae6-488b-b883-35455abb3e0b" containerName="revoke-keys"
Nov 02 17:45:17.360339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.360327    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be480d4-5ae6-488b-b883-35455abb3e0b" containerName="revoke-keys"
Nov 02 17:45:17.365121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.365099    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.370269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.370246    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:17.383188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.383162    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"]
Nov 02 17:45:17.402015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.401942    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.402015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.401983    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnfm\" (UniqueName: \"kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.402231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.402079    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.503992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.503958    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.504176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.504016    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnfm\" (UniqueName: \"kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.504176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.504076    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.504720 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.504695    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.506942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.506916    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.513321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.513293    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnfm\" (UniqueName: \"kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm\") pod \"maas-api-revoke-sub-d7d08855-hjb4c\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.540289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.540265    5042 scope.go:122] "RemoveContainer" containerID="3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed"
Nov 02 17:45:17.683563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.683469    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:17.991143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:17.991117    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"]
Nov 02 17:45:17.992755 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:17.992732    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4465b7_fd5c_4511_8a30_57f3f52ffc1d.slice/crio-5e370041e82094207b43787b3210ac323c9ff458cc890b010460f4b7492d1741 WatchSource:0}: Error finding container 5e370041e82094207b43787b3210ac323c9ff458cc890b010460f4b7492d1741: Status 404 returned error can't find the container with id 5e370041e82094207b43787b3210ac323c9ff458cc890b010460f4b7492d1741
Nov 02 17:45:18.257938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.247294    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c" event={"ID":"da4465b7-fd5c-4511-8a30-57f3f52ffc1d","Type":"ContainerStarted","Data":"5e370041e82094207b43787b3210ac323c9ff458cc890b010460f4b7492d1741"}
Nov 02 17:45:18.320862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.319042    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"]
Nov 02 17:45:18.328914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.328875    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerStarted","Data":"97d479fc62bbf6e69f7b1760b3af55d6c1b112a68bef329fc663437d50cccb19"}
Nov 02 17:45:18.336432 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.336396    5042 generic.go:356] "Generic (PLEG): container finished" podID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerID="ae42f994324c7fd6a03f1012f15489fe325006deb381352238aaab85a5b0649c" exitCode=60
Nov 02 17:45:18.336538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.336462    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerDied","Data":"ae42f994324c7fd6a03f1012f15489fe325006deb381352238aaab85a5b0649c"}
Nov 02 17:45:18.336710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.336534    5042 scope.go:122] "RemoveContainer" containerID="d102fe56c2c40c025549f44687d0170d7aa9678ee2e5d97fd2e6dadc1700c5d2"
Nov 02 17:45:18.611989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.611909    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"]
Nov 02 17:45:18.619235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.619206    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.627288 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.627230    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"]
Nov 02 17:45:18.659665 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:18.659629    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:45:18.671817 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.671790    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:18.675777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.675756    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:18.811743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.811706    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x\" (UniqueName: \"kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x\") pod \"f529fc7a-652f-4b7c-8581-14537f0438e0\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") "
Nov 02 17:45:18.812409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.812361    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq\" (UniqueName: \"kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq\") pod \"0360c96d-a7ea-4ac7-9424-26ae315830dc\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") "
Nov 02 17:45:18.812543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.812427    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca\") pod \"0360c96d-a7ea-4ac7-9424-26ae315830dc\" (UID: \"0360c96d-a7ea-4ac7-9424-26ae315830dc\") "
Nov 02 17:45:18.812543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.812483    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca\") pod \"f529fc7a-652f-4b7c-8581-14537f0438e0\" (UID: \"f529fc7a-652f-4b7c-8581-14537f0438e0\") "
Nov 02 17:45:18.813053 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.813025    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klbp4\" (UniqueName: \"kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.813211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.813121    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca" pod "f529fc7a-652f-4b7c-8581-14537f0438e0" (UID: "f529fc7a-652f-4b7c-8581-14537f0438e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:18.813315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.813227    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.813315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.813284    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.813500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.813473    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f529fc7a-652f-4b7c-8581-14537f0438e0-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.813935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.813826    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca" pod "0360c96d-a7ea-4ac7-9424-26ae315830dc" (UID: "0360c96d-a7ea-4ac7-9424-26ae315830dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:18.814664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.814631    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x" pod "f529fc7a-652f-4b7c-8581-14537f0438e0" (UID: "f529fc7a-652f-4b7c-8581-14537f0438e0"). InnerVolumeSpecName "kube-api-access-4b94x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:18.815505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.815474    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq" pod "0360c96d-a7ea-4ac7-9424-26ae315830dc" (UID: "0360c96d-a7ea-4ac7-9424-26ae315830dc"). InnerVolumeSpecName "kube-api-access-jhvrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:18.915395 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.915296    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klbp4\" (UniqueName: \"kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.915553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.915407    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.915553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.915457    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.915705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.915692    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvrq\" (UniqueName: \"kubernetes.io/projected/0360c96d-a7ea-4ac7-9424-26ae315830dc-kube-api-access-jhvrq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.915754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.915710    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0360c96d-a7ea-4ac7-9424-26ae315830dc-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.915754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.915727    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b94x\" (UniqueName: \"kubernetes.io/projected/f529fc7a-652f-4b7c-8581-14537f0438e0-kube-api-access-4b94x\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:18.916191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.916161    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.918105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.918075    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.923738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.923719    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klbp4\" (UniqueName: \"kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4\") pod \"maas-api-revoke-sub-a983067e-xpvf2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:18.969417 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:18.969385    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:19.274510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.274483    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"]
Nov 02 17:45:19.276445 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:19.276417    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd425fb7_693f_43ea_836d_a5575d4b17c2.slice/crio-b2e4d863cd39a218a7b5453cbfde63cbc684d8cbd151147564692d9e97ad445c WatchSource:0}: Error finding container b2e4d863cd39a218a7b5453cbfde63cbc684d8cbd151147564692d9e97ad445c: Status 404 returned error can't find the container with id b2e4d863cd39a218a7b5453cbfde63cbc684d8cbd151147564692d9e97ad445c
Nov 02 17:45:19.345570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.345540    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"
Nov 02 17:45:19.345745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.345537    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58" event={"ID":"f529fc7a-652f-4b7c-8581-14537f0438e0","Type":"ContainerDied","Data":"8a94e3a7c4054d48192787ea1c2a5692e3003d920da42489f2967cc0529ff8b5"}
Nov 02 17:45:19.345745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.345667    5042 scope.go:122] "RemoveContainer" containerID="f50345578cc50ac7a5cb4608a68ed549f35948844bd77744e01bdd47e9e2652c"
Nov 02 17:45:19.347049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.347031    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"
Nov 02 17:45:19.347339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.347093    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj" event={"ID":"0360c96d-a7ea-4ac7-9424-26ae315830dc","Type":"ContainerDied","Data":"2b2afcbd83fd9d8d137c80b93e30bfe1c31c4dcc71826d81865081807269d392"}
Nov 02 17:45:19.349161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.349135    5042 generic.go:356] "Generic (PLEG): container finished" podID="da4465b7-fd5c-4511-8a30-57f3f52ffc1d" containerID="ceb549d576e7c0f4ddca390d87fb3e77ff333fb897b28eb1bbb087f964f14087" exitCode=0
Nov 02 17:45:19.349271 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.349192    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c" event={"ID":"da4465b7-fd5c-4511-8a30-57f3f52ffc1d","Type":"ContainerDied","Data":"ceb549d576e7c0f4ddca390d87fb3e77ff333fb897b28eb1bbb087f964f14087"}
Nov 02 17:45:19.350669 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.350647    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2" event={"ID":"dd425fb7-693f-43ea-836d-a5575d4b17c2","Type":"ContainerStarted","Data":"b2e4d863cd39a218a7b5453cbfde63cbc684d8cbd151147564692d9e97ad445c"}
Nov 02 17:45:19.352775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.352748    5042 generic.go:356] "Generic (PLEG): container finished" podID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerID="97d479fc62bbf6e69f7b1760b3af55d6c1b112a68bef329fc663437d50cccb19" exitCode=60
Nov 02 17:45:19.352918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.352772    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerDied","Data":"97d479fc62bbf6e69f7b1760b3af55d6c1b112a68bef329fc663437d50cccb19"}
Nov 02 17:45:19.353249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.353231    5042 scope.go:122] "RemoveContainer" containerID="97d479fc62bbf6e69f7b1760b3af55d6c1b112a68bef329fc663437d50cccb19"
Nov 02 17:45:19.353556 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:19.353530    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv_odh-ai-gateway-infra(7780aee0-96a8-4ed3-bb27-cab9dffe3a04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04"
Nov 02 17:45:19.357022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.357007    5042 scope.go:122] "RemoveContainer" containerID="ae42f994324c7fd6a03f1012f15489fe325006deb381352238aaab85a5b0649c"
Nov 02 17:45:19.364496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.364472    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"]
Nov 02 17:45:19.369856 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.369051    5042 scope.go:122] "RemoveContainer" containerID="3d933449952ba5762a6cb054fa746d785e49d2952561416f18fbdf8271d982ed"
Nov 02 17:45:19.396682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.396631    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"]
Nov 02 17:45:19.402545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.402514    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894025-nd7nj"]
Nov 02 17:45:19.430363 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.430333    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"]
Nov 02 17:45:19.434749 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.434714    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-xqq58"]
Nov 02 17:45:19.536243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.536171    5042 scope.go:122] "RemoveContainer" containerID="4b5b2acea64acebc6fd0c66b310da6fbe6f59dedc37b53c4232df11ca5cc75f0"
Nov 02 17:45:19.545185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.545160    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" path="/var/lib/kubelet/pods/0360c96d-a7ea-4ac7-9424-26ae315830dc/volumes"
Nov 02 17:45:19.545576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:19.545548    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" path="/var/lib/kubelet/pods/f529fc7a-652f-4b7c-8581-14537f0438e0/volumes"
Nov 02 17:45:20.363638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.363601    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerStarted","Data":"b1bf722cbd63bae194f66f5a705cc11c9bdb6d94a1dea345e5ecc2fd7bc76abb"}
Nov 02 17:45:20.368169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.368136    5042 generic.go:356] "Generic (PLEG): container finished" podID="dd425fb7-693f-43ea-836d-a5575d4b17c2" containerID="6433d99ef8a2e5272db048edab7d2fdcfde36ca7a2e7a1f152bd414c900811ed" exitCode=0
Nov 02 17:45:20.368291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.368168    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2" event={"ID":"dd425fb7-693f-43ea-836d-a5575d4b17c2","Type":"ContainerDied","Data":"6433d99ef8a2e5272db048edab7d2fdcfde36ca7a2e7a1f152bd414c900811ed"}
Nov 02 17:45:20.567631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.567598    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"]
Nov 02 17:45:20.670888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.670831    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:20.678017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.677992    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:20.742493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.742458    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca\") pod \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") "
Nov 02 17:45:20.742682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.742560    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca\") pod \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") "
Nov 02 17:45:20.742682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.742638    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca\") pod \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") "
Nov 02 17:45:20.742796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.742682    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw\" (UniqueName: \"kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw\") pod \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\" (UID: \"7780aee0-96a8-4ed3-bb27-cab9dffe3a04\") "
Nov 02 17:45:20.742796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.742724    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm\" (UniqueName: \"kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm\") pod \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\" (UID: \"da4465b7-fd5c-4511-8a30-57f3f52ffc1d\") "
Nov 02 17:45:20.743143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.743114    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca" pod "7780aee0-96a8-4ed3-bb27-cab9dffe3a04" (UID: "7780aee0-96a8-4ed3-bb27-cab9dffe3a04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:20.743356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.743164    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca" pod "da4465b7-fd5c-4511-8a30-57f3f52ffc1d" (UID: "da4465b7-fd5c-4511-8a30-57f3f52ffc1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:20.743898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.743875    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:20.744002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.743903    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:20.745816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.745775    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw" pod "7780aee0-96a8-4ed3-bb27-cab9dffe3a04" (UID: "7780aee0-96a8-4ed3-bb27-cab9dffe3a04"). InnerVolumeSpecName "kube-api-access-6pndw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:20.745961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.745925    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm" pod "da4465b7-fd5c-4511-8a30-57f3f52ffc1d" (UID: "da4465b7-fd5c-4511-8a30-57f3f52ffc1d"). InnerVolumeSpecName "kube-api-access-pfnfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:20.746031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.745958    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca" pod "da4465b7-fd5c-4511-8a30-57f3f52ffc1d" (UID: "da4465b7-fd5c-4511-8a30-57f3f52ffc1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:20.844895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.844793    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pndw\" (UniqueName: \"kubernetes.io/projected/7780aee0-96a8-4ed3-bb27-cab9dffe3a04-kube-api-access-6pndw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:20.844895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.844823    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfnfm\" (UniqueName: \"kubernetes.io/projected/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-kube-api-access-pfnfm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:20.844895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:20.844833    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4465b7-fd5c-4511-8a30-57f3f52ffc1d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:21.379044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.379007    5042 generic.go:356] "Generic (PLEG): container finished" podID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerID="b1bf722cbd63bae194f66f5a705cc11c9bdb6d94a1dea345e5ecc2fd7bc76abb" exitCode=60
Nov 02 17:45:21.379525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.379096    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerDied","Data":"b1bf722cbd63bae194f66f5a705cc11c9bdb6d94a1dea345e5ecc2fd7bc76abb"}
Nov 02 17:45:21.379525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.379217    5042 scope.go:122] "RemoveContainer" containerID="4b5b2acea64acebc6fd0c66b310da6fbe6f59dedc37b53c4232df11ca5cc75f0"
Nov 02 17:45:21.382747 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.382709    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c" event={"ID":"da4465b7-fd5c-4511-8a30-57f3f52ffc1d","Type":"ContainerDied","Data":"5e370041e82094207b43787b3210ac323c9ff458cc890b010460f4b7492d1741"}
Nov 02 17:45:21.382887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.382750    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"
Nov 02 17:45:21.383001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.382773    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e370041e82094207b43787b3210ac323c9ff458cc890b010460f4b7492d1741"
Nov 02 17:45:21.384297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.384276    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"
Nov 02 17:45:21.384297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.384284    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv" event={"ID":"7780aee0-96a8-4ed3-bb27-cab9dffe3a04","Type":"ContainerDied","Data":"81eb28dc88b8af2c5904c923ce410ded4e7749895b24709c2b9f7eab3bd53c9c"}
Nov 02 17:45:21.405764 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.405740    5042 scope.go:122] "RemoveContainer" containerID="97d479fc62bbf6e69f7b1760b3af55d6c1b112a68bef329fc663437d50cccb19"
Nov 02 17:45:21.433052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.433020    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"]
Nov 02 17:45:21.443462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.443433    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894025-4rxfv"]
Nov 02 17:45:21.550180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.550146    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" path="/var/lib/kubelet/pods/7780aee0-96a8-4ed3-bb27-cab9dffe3a04/volumes"
Nov 02 17:45:21.601662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.601638    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:21.604061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.604037    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:21.654348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.654266    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76\" (UniqueName: \"kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76\") pod \"669d4462-23c5-4b90-b3c0-4175661fbec3\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") "
Nov 02 17:45:21.654514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.654428    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4\" (UniqueName: \"kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4\") pod \"dd425fb7-693f-43ea-836d-a5575d4b17c2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") "
Nov 02 17:45:21.654577 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.654503    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca\") pod \"dd425fb7-693f-43ea-836d-a5575d4b17c2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") "
Nov 02 17:45:21.654577 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.654563    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca\") pod \"669d4462-23c5-4b90-b3c0-4175661fbec3\" (UID: \"669d4462-23c5-4b90-b3c0-4175661fbec3\") "
Nov 02 17:45:21.654682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.654617    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca\") pod \"dd425fb7-693f-43ea-836d-a5575d4b17c2\" (UID: \"dd425fb7-693f-43ea-836d-a5575d4b17c2\") "
Nov 02 17:45:21.655379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.655250    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca" pod "669d4462-23c5-4b90-b3c0-4175661fbec3" (UID: "669d4462-23c5-4b90-b3c0-4175661fbec3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:21.656354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.656326    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca" pod "dd425fb7-693f-43ea-836d-a5575d4b17c2" (UID: "dd425fb7-693f-43ea-836d-a5575d4b17c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:21.657300 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.657262    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76" pod "669d4462-23c5-4b90-b3c0-4175661fbec3" (UID: "669d4462-23c5-4b90-b3c0-4175661fbec3"). InnerVolumeSpecName "kube-api-access-9tn76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:21.658435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.658411    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca" pod "dd425fb7-693f-43ea-836d-a5575d4b17c2" (UID: "dd425fb7-693f-43ea-836d-a5575d4b17c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:21.658566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.658546    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4" pod "dd425fb7-693f-43ea-836d-a5575d4b17c2" (UID: "dd425fb7-693f-43ea-836d-a5575d4b17c2"). InnerVolumeSpecName "kube-api-access-klbp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:21.756868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.756801    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klbp4\" (UniqueName: \"kubernetes.io/projected/dd425fb7-693f-43ea-836d-a5575d4b17c2-kube-api-access-klbp4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:21.756868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.756832    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:21.756868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.756862    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/669d4462-23c5-4b90-b3c0-4175661fbec3-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:21.756868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.756872    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd425fb7-693f-43ea-836d-a5575d4b17c2-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:21.757173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:21.756881    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tn76\" (UniqueName: \"kubernetes.io/projected/669d4462-23c5-4b90-b3c0-4175661fbec3-kube-api-access-9tn76\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:22.398868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.394515    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"
Nov 02 17:45:22.398868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.394529    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2" event={"ID":"dd425fb7-693f-43ea-836d-a5575d4b17c2","Type":"ContainerDied","Data":"b2e4d863cd39a218a7b5453cbfde63cbc684d8cbd151147564692d9e97ad445c"}
Nov 02 17:45:22.398868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.394576    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2e4d863cd39a218a7b5453cbfde63cbc684d8cbd151147564692d9e97ad445c"
Nov 02 17:45:22.398868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.397283    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"
Nov 02 17:45:22.398868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.397309    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd" event={"ID":"669d4462-23c5-4b90-b3c0-4175661fbec3","Type":"ContainerDied","Data":"c25cac06c3fbbf9a8843303463dd5b75d5f58a6cd89e07e20f2b406aa9988e14"}
Nov 02 17:45:22.398868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.397361    5042 scope.go:122] "RemoveContainer" containerID="b1bf722cbd63bae194f66f5a705cc11c9bdb6d94a1dea345e5ecc2fd7bc76abb"
Nov 02 17:45:22.431176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.431133    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"]
Nov 02 17:45:22.436464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:22.436433    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894025-r5qdd"]
Nov 02 17:45:23.558120 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:23.557566    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" path="/var/lib/kubelet/pods/669d4462-23c5-4b90-b3c0-4175661fbec3/volumes"
Nov 02 17:45:24.981856 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:24.980036    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:24.981856 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:24.980913    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.072866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:25.071339    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:45:25.167193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:25.164934    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.203038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:25.202987    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.220997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:25.220947    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.984063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:25.983996    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.987492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:25.985766    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.007458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.007354    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:45:26.008465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.008444    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:45:26.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.072772    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.075866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.073295    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:45:26.182866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.174227    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.182866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.174982    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:45:26.354295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.327071    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:45:26.381866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.350973    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.383258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.377306    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:45:26.383493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.383477    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:45:26.495863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:26.468007    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:45:27.134361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.088078    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:45:27.134361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.102009    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:27.134361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.102931    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:45:27.187861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.183016    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:27.425927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.418450    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:45:27.501914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.481260    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:45:27.599891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.481551    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:45:27.901970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.901921    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:27.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:27.952027    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:45:28.162638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.162521    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:45:28.208072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.165725    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:45:28.208886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.165758    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:45:28.312879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.201984    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:28.518905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.502059    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:28.599996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.599952    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:28.624866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.601271    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:45:28.978866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.970302    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:28.985428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:28.985195    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:45:29.158227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.158200    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:45:29.230091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.229986    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:45:29.232879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.230690    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:45:29.263000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.262036    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:45:29.325864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.321967    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:29.389976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.389928    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:45:29.532184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.532088    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:29.606466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.606406    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:29.739178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.738095    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:45:29.780036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.780005    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:45:29.981276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:29.980115    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:45:31.617658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.617625    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"]
Nov 02 17:45:31.618169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618155    5042 cpu_manager.go:414] "containerMap: removing container" podUID="da4465b7-fd5c-4511-8a30-57f3f52ffc1d" containerName="revoke-keys"
Nov 02 17:45:31.618169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618169    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4465b7-fd5c-4511-8a30-57f3f52ffc1d" containerName="revoke-keys"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618178    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dd425fb7-693f-43ea-836d-a5575d4b17c2" containerName="revoke-keys"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618183    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd425fb7-693f-43ea-836d-a5575d4b17c2" containerName="revoke-keys"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618193    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618200    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618206    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618210    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618217    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618222    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618234    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618238    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618247    5042 cpu_manager.go:414] "containerMap: removing container" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618251    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618264    5042 cpu_manager.go:414] "containerMap: removing container" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618268    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618280    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.618283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618284    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618296    5042 cpu_manager.go:414] "containerMap: removing container" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618300    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618309    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618314    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618320    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618324    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618400    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618410    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618417    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618425    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618429    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618435    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618440    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4465b7-fd5c-4511-8a30-57f3f52ffc1d" containerName="revoke-keys"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618446    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd425fb7-693f-43ea-836d-a5575d4b17c2" containerName="revoke-keys"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618452    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618457    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618462    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:31.619004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.618470    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:31.621920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.621902    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.641646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.641619    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"]
Nov 02 17:45:31.642742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.642718    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:45:31.642944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.642729    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:45:31.642944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.642750    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:31.642944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.642786    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:45:31.659415 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.659382    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvfr\" (UniqueName: \"kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.659579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.659476    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.659624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.659574    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.760777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.760731    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvfr\" (UniqueName: \"kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.760960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.760812    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.760960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.760852    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.761506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.761488    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.763440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.763413    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.774889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.774865    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvfr\" (UniqueName: \"kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr\") pod \"maas-api-revoke-sub-e09e6875-tztm6\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:31.935619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:31.935535    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:32.224248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:32.224218    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"]
Nov 02 17:45:32.225284 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:32.225259    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955d6a99_dd07_47e9_af0d_4afd4f515be5.slice/crio-8f925c970bd7ce2b09e360bd25a3037082ca82b7f0aa207ab33ae6965ea0ab51 WatchSource:0}: Error finding container 8f925c970bd7ce2b09e360bd25a3037082ca82b7f0aa207ab33ae6965ea0ab51: Status 404 returned error can't find the container with id 8f925c970bd7ce2b09e360bd25a3037082ca82b7f0aa207ab33ae6965ea0ab51
Nov 02 17:45:33.203292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:33.203256    5042 generic.go:356] "Generic (PLEG): container finished" podID="955d6a99-dd07-47e9-af0d-4afd4f515be5" containerID="a03154a250270a386f4d1086cd14f65488c8c55fd352972ec52036b6a56c2652" exitCode=0
Nov 02 17:45:33.204293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:33.203965    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6" event={"ID":"955d6a99-dd07-47e9-af0d-4afd4f515be5","Type":"ContainerDied","Data":"a03154a250270a386f4d1086cd14f65488c8c55fd352972ec52036b6a56c2652"}
Nov 02 17:45:33.204554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:33.204532    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6" event={"ID":"955d6a99-dd07-47e9-af0d-4afd4f515be5","Type":"ContainerStarted","Data":"8f925c970bd7ce2b09e360bd25a3037082ca82b7f0aa207ab33ae6965ea0ab51"}
Nov 02 17:45:34.010478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.010446    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"]
Nov 02 17:45:34.011056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.011040    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:34.011112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.011057    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f529fc7a-652f-4b7c-8581-14537f0438e0" containerName="cleanup"
Nov 02 17:45:34.011112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.011083    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:34.011112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.011088    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780aee0-96a8-4ed3-bb27-cab9dffe3a04" containerName="cleanup"
Nov 02 17:45:34.011215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.011199    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="669d4462-23c5-4b90-b3c0-4175661fbec3" containerName="cleanup"
Nov 02 17:45:34.011215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.011210    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0360c96d-a7ea-4ac7-9424-26ae315830dc" containerName="cleanup"
Nov 02 17:45:34.014694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.014675    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.026879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.024180    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"]
Nov 02 17:45:34.086392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.086352    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.086603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.086580    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.086718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.086701    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8prk2\" (UniqueName: \"kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.188219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.188179    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.188399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.188275    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8prk2\" (UniqueName: \"kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.188399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.188312    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.188880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.188859    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.191167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.191143    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.197208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.197184    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8prk2\" (UniqueName: \"kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2\") pod \"maas-api-revoke-sub-1f82bf2f-zmrdl\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.332133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.332091    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:34.408924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.406990    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:34.492215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.492178    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca\") pod \"955d6a99-dd07-47e9-af0d-4afd4f515be5\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") "
Nov 02 17:45:34.492399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.492249    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr\" (UniqueName: \"kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr\") pod \"955d6a99-dd07-47e9-af0d-4afd4f515be5\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") "
Nov 02 17:45:34.492399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.492330    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca\") pod \"955d6a99-dd07-47e9-af0d-4afd4f515be5\" (UID: \"955d6a99-dd07-47e9-af0d-4afd4f515be5\") "
Nov 02 17:45:34.492583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.492561    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca" pod "955d6a99-dd07-47e9-af0d-4afd4f515be5" (UID: "955d6a99-dd07-47e9-af0d-4afd4f515be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:34.493010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.492915    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:34.494626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.494596    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca" pod "955d6a99-dd07-47e9-af0d-4afd4f515be5" (UID: "955d6a99-dd07-47e9-af0d-4afd4f515be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:34.494626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.494619    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr" pod "955d6a99-dd07-47e9-af0d-4afd4f515be5" (UID: "955d6a99-dd07-47e9-af0d-4afd4f515be5"). InnerVolumeSpecName "kube-api-access-ntvfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:34.594030 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.593939    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntvfr\" (UniqueName: \"kubernetes.io/projected/955d6a99-dd07-47e9-af0d-4afd4f515be5-kube-api-access-ntvfr\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:34.594030 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.593977    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/955d6a99-dd07-47e9-af0d-4afd4f515be5-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:34.617160 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:34.617129    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c3ea23_26d7_418c_8375_e7e9f959b7c3.slice/crio-e6a141914a7c932df14dc796faf4ffd6a1faf0b261f2278eeb7fdf16e4f0e2fc WatchSource:0}: Error finding container e6a141914a7c932df14dc796faf4ffd6a1faf0b261f2278eeb7fdf16e4f0e2fc: Status 404 returned error can't find the container with id e6a141914a7c932df14dc796faf4ffd6a1faf0b261f2278eeb7fdf16e4f0e2fc
Nov 02 17:45:34.618826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:34.618766    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"]
Nov 02 17:45:35.224034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.223996    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl" event={"ID":"65c3ea23-26d7-418c-8375-e7e9f959b7c3","Type":"ContainerStarted","Data":"c33de816728db41e86a96ecb9b1009a19bc7df9ca39f2c342d2646433516fc94"}
Nov 02 17:45:35.224217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.224060    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl" event={"ID":"65c3ea23-26d7-418c-8375-e7e9f959b7c3","Type":"ContainerStarted","Data":"e6a141914a7c932df14dc796faf4ffd6a1faf0b261f2278eeb7fdf16e4f0e2fc"}
Nov 02 17:45:35.225713 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.225693    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"
Nov 02 17:45:35.225867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.225686    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6" event={"ID":"955d6a99-dd07-47e9-af0d-4afd4f515be5","Type":"ContainerDied","Data":"8f925c970bd7ce2b09e360bd25a3037082ca82b7f0aa207ab33ae6965ea0ab51"}
Nov 02 17:45:35.225867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.225823    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f925c970bd7ce2b09e360bd25a3037082ca82b7f0aa207ab33ae6965ea0ab51"
Nov 02 17:45:35.244278 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.244237    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl" podStartSLOduration=2.24422428 podStartE2EDuration="2.24422428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:35.241274528 +0000 UTC m=+747.820683914" watchObservedRunningTime="2026-11-02 17:45:35.24422428 +0000 UTC m=+747.823633705"
Nov 02 17:45:35.515642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.515596    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"]
Nov 02 17:45:35.516479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.516456    5042 cpu_manager.go:414] "containerMap: removing container" podUID="955d6a99-dd07-47e9-af0d-4afd4f515be5" containerName="revoke-keys"
Nov 02 17:45:35.516479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.516475    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955d6a99-dd07-47e9-af0d-4afd4f515be5" containerName="revoke-keys"
Nov 02 17:45:35.516670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.516623    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="955d6a99-dd07-47e9-af0d-4afd4f515be5" containerName="revoke-keys"
Nov 02 17:45:35.521170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.521145    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.551883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.551819    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"]
Nov 02 17:45:35.608309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608265    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608352    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608400    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608519    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608561    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608720 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608701    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608748    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608787    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nl2\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.608983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.608965    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710283    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710345    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710377    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710406    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710492    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710510    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710548    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710597    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.710827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710633    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nl2\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.711038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.710925    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.711359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.711330    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.711770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.711723    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.712028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.712003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.712176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.712141    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.713618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.713591    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.713725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.713664    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.727627 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.727598    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nl2\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.728711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.728689    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:35.851731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:35.851647    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:36.148972 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.148947    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"]
Nov 02 17:45:36.149101 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:36.148972    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9129ca9b_5439_40df_b2a6_a2bfbadc4740.slice/crio-a8001f208ca5f4211f306ccaa898ac0580b2b4e07530ef5126e626b7e16b9cd4 WatchSource:0}: Error finding container a8001f208ca5f4211f306ccaa898ac0580b2b4e07530ef5126e626b7e16b9cd4: Status 404 returned error can't find the container with id a8001f208ca5f4211f306ccaa898ac0580b2b4e07530ef5126e626b7e16b9cd4
Nov 02 17:45:36.151482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.151453    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:36.151528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.151517    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:36.151568 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.151545    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:36.234915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.234883    5042 generic.go:356] "Generic (PLEG): container finished" podID="65c3ea23-26d7-418c-8375-e7e9f959b7c3" containerID="c33de816728db41e86a96ecb9b1009a19bc7df9ca39f2c342d2646433516fc94" exitCode=0
Nov 02 17:45:36.235054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.234963    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl" event={"ID":"65c3ea23-26d7-418c-8375-e7e9f959b7c3","Type":"ContainerDied","Data":"c33de816728db41e86a96ecb9b1009a19bc7df9ca39f2c342d2646433516fc94"}
Nov 02 17:45:36.237106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:36.237085    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" event={"ID":"9129ca9b-5439-40df-b2a6-a2bfbadc4740","Type":"ContainerStarted","Data":"a8001f208ca5f4211f306ccaa898ac0580b2b4e07530ef5126e626b7e16b9cd4"}
Nov 02 17:45:37.017546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.017509    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"]
Nov 02 17:45:37.022530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.022494    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.035735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.035711    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"]
Nov 02 17:45:37.137507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137463    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.137702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137514    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.137702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137552    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.137702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137588    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.137874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137831    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.137957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137939    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.138001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.137971    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.138065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.138044    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.138185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.138164    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfcsh\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239254 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239215    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239262    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239295    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239319    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239478    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239536    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239560    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239592    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.239777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.239688    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfcsh\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.240540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.240513    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.240730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.240521    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.240827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.240786    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.241740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.241715    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.242280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.242255    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.243399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.243383    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.247551 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.247529    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.251249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.251173    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" event={"ID":"9129ca9b-5439-40df-b2a6-a2bfbadc4740","Type":"ContainerStarted","Data":"68b4f358aa3e11b84dd66c474ab7a8af24b4c24749a0ddce2dccdc89ae8afb3a"}
Nov 02 17:45:37.251538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.251515    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfcsh\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.251975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.251778    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.321127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.300804    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podStartSLOduration=2.300782293 podStartE2EDuration="2.300782293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:37.295675296 +0000 UTC m=+749.875084682" watchObservedRunningTime="2026-11-02 17:45:37.300782293 +0000 UTC m=+749.880191680"
Nov 02 17:45:37.339338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.339307    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:37.551023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.550990    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:37.646048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.645958    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca\") pod \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") "
Nov 02 17:45:37.646048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.646003    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2\" (UniqueName: \"kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2\") pod \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") "
Nov 02 17:45:37.646289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.646155    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca\") pod \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\" (UID: \"65c3ea23-26d7-418c-8375-e7e9f959b7c3\") "
Nov 02 17:45:37.646394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.646365    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca" pod "65c3ea23-26d7-418c-8375-e7e9f959b7c3" (UID: "65c3ea23-26d7-418c-8375-e7e9f959b7c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:37.646812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.646738    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:37.648195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.648173    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2" pod "65c3ea23-26d7-418c-8375-e7e9f959b7c3" (UID: "65c3ea23-26d7-418c-8375-e7e9f959b7c3"). InnerVolumeSpecName "kube-api-access-8prk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:37.648302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.648288    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca" pod "65c3ea23-26d7-418c-8375-e7e9f959b7c3" (UID: "65c3ea23-26d7-418c-8375-e7e9f959b7c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:37.748029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.747990    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c3ea23-26d7-418c-8375-e7e9f959b7c3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:37.748271 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.748253    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8prk2\" (UniqueName: \"kubernetes.io/projected/65c3ea23-26d7-418c-8375-e7e9f959b7c3-kube-api-access-8prk2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:37.835633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.835593    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"]
Nov 02 17:45:37.836168 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.836119    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:37.836307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.836261    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:37.836422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.836399    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:37.853645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.853433    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:37.855265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.855239    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:37.855483 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:37.855451    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:38.262217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.262186    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" event={"ID":"22cced10-bb3b-4b8e-8de0-f22ea4c39121","Type":"ContainerStarted","Data":"392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b"}
Nov 02 17:45:38.265703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.262237    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" event={"ID":"22cced10-bb3b-4b8e-8de0-f22ea4c39121","Type":"ContainerStarted","Data":"903d598866869779ef2f521596edd879d34f0a896bd8576442e4f6ce76083805"}
Nov 02 17:45:38.265703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.263729    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl" event={"ID":"65c3ea23-26d7-418c-8375-e7e9f959b7c3","Type":"ContainerDied","Data":"e6a141914a7c932df14dc796faf4ffd6a1faf0b261f2278eeb7fdf16e4f0e2fc"}
Nov 02 17:45:38.265703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.263757    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"
Nov 02 17:45:38.265703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.263767    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a141914a7c932df14dc796faf4ffd6a1faf0b261f2278eeb7fdf16e4f0e2fc"
Nov 02 17:45:38.315778 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.315716    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podStartSLOduration=2.31570354 podStartE2EDuration="2.31570354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:38.304271407 +0000 UTC m=+750.883680792" watchObservedRunningTime="2026-11-02 17:45:38.31570354 +0000 UTC m=+750.895112925"
Nov 02 17:45:38.339993 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.339955    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:38.341637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.341610    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:38.341795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.341658    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:38.852002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.851963    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:38.852184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:38.852025    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:39.341864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:39.340561    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:39.341864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:39.340619    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:39.852145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:39.852111    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:39.852317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:39.852165    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:40.340350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:40.340301    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:40.340555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:40.340362    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:40.860862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:40.860227    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:40.860862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:40.860290    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:41.393372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:41.393335    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:41.393689 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:41.393659    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:41.788865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:41.787936    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:41.860865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:41.860229    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:41.860865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:41.860294    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:41.997014 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:41.996961    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.196861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.195982    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.217863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.216929    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Nov 02 17:45:42.218173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.218138    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.238864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.235854    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.275868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.271881    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"]
Nov 02 17:45:42.290865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.287718    5042 cpu_manager.go:414] "containerMap: removing container" podUID="65c3ea23-26d7-418c-8375-e7e9f959b7c3" containerName="revoke-keys"
Nov 02 17:45:42.290865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.287747    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c3ea23-26d7-418c-8375-e7e9f959b7c3" containerName="revoke-keys"
Nov 02 17:45:42.290865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.287914    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c3ea23-26d7-418c-8375-e7e9f959b7c3" containerName="revoke-keys"
Nov 02 17:45:42.291322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.291278    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Nov 02 17:45:42.292550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.292512    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.306488 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.306456    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"]
Nov 02 17:45:42.401990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.354822    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.415982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.415940    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2c9\" (UniqueName: \"kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.439863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.416343    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.440258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.368028    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:45:42.440408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.379347    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:42.440552 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.440527    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:42.440650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.367407    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:42.443899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.367360    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:45:42.511104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.511014    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:45:42.568864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.523917    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.597875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.596338    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.597875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.596518    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.597875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.596552    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2c9\" (UniqueName: \"kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.597875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.597658    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.674920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.647980    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.828923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.820948    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2c9\" (UniqueName: \"kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.851454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.851413    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-849cf249-ssnnq\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:42.874484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.855195    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:42.886866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.875111    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:42.887162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.873937    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:42.889177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:42.887648    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:45:43.011215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.002940    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:43.011215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.003768    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:45:43.079918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.042436    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:43.202782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.202730    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:43.203980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.203948    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:45:43.246876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.243945    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:43.246876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.244333    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:45:43.246876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.244638    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:45:43.246876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.244715    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:45:43.407222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.368985    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:43.407222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.369043    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:43.586007 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.585954    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:43.723957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.697983    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:45:43.901223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.901182    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:45:43.906981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.906953    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:43.932898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.932833    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:43.960862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.959504    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:45:44.064035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.063929    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:44.107934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:43.959575    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:45:44.276990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.276922    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:44.300061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.300019    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:44.328964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.327185    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:44.681069 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.680968    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:44.681371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.681321    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:44.736075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:44.732956    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.031866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.027484    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:45.164873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.038005    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:45:45.164873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.084545    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.294922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.294099    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.528875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.525896    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.528875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.164245    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:45.538864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.534291    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:45.538864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.534342    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.538864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.536052    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:45.538864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.536092    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.584276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.583933    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:45.612868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.585960    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:45.612868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.586009    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:45.677975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.675689    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:45.677975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.675748    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:45.685431 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.679196    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:45:45.833125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.823022    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:45.833125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.828977    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:45.833125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:45.829032    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:46.052162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.052125    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:46.053523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.053484    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:46.168546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.168500    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:45:46.239639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.239606    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"]
Nov 02 17:45:46.372451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.372415    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"]
Nov 02 17:45:46.372698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.372682    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"]
Nov 02 17:45:46.372929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.372910    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:46.488951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.488905    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:46.489241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.489203    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:46.763258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.743704    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b7gw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:46.763258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.743772    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:46.763258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.744189    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:46.799851 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.770790    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:45:46.799851 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.773012    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:46.799851 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.788013    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2w9k\" (UniqueName: \"kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:46.799851 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.788154    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:46.799851 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.788369    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:46.799851 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.790369    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:46.826907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.825946    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:46.929924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.692962    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:46.995961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:46.995918    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.041873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.021082    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.058045 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.058013    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2w9k\" (UniqueName: \"kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.154143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.092286    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.195768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.170229    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:47.195768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.170295    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:47.195768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.173860    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.195768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.187951    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:47.214718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.214684    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2w9k\" (UniqueName: \"kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k\") pod \"maas-api-revoke-sub-8a112fd1-f2r28\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.224491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.217209    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq" event={"ID":"2805413d-aa54-45dc-826d-c68d83d19eba","Type":"ContainerStarted","Data":"48e7ee7e12bfc202ce3a9408b92ede127f181a6bbcce27e0194d82c84aea4f4e"}
Nov 02 17:45:47.224491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.218587    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:47.297726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.237002    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:47.297726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.237069    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:47.354894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.341356    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:47.354894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.341418    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:47.749920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.749788    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:45:47.772068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.770994    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:47.884482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.801559    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:47.884482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.801923    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:45:47.884482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.832671    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:45:47.912105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.909692    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:45:47.912105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.912013    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:47.912673 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.912650    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Nov 02 17:45:47.912877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:47.912826    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Nov 02 17:45:48.071990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.071887    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:45:48.233434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.223610    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:45:48.484383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.432046    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:48.484383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.432109    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:48.484383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.463070    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:48.484383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.463128    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:48.517210 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.504472    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:45:48.549895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.547337    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:45:48.556945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.552480    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:45:48.655744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.653910    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:45:48.727074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.727042    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:45:48.745044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.745006    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"]
Nov 02 17:45:48.868038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.868002    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:48.995323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:48.995276    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq" event={"ID":"2805413d-aa54-45dc-826d-c68d83d19eba","Type":"ContainerStarted","Data":"29be0999b77d1b7e8aa0983860a1f96dd2b656de5be9bee48fa0afb59329eec9"}
Nov 02 17:45:49.063060 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:45:49.062808    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:45:49.340757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:49.340663    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 02 17:45:49.340757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:49.340724    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 02 17:45:49.497054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:49.495480    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:49.497054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:49.496989    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:49.497960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:49.497924    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 17:45:50.010284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.010246    5042 generic.go:356] "Generic (PLEG): container finished" podID="2805413d-aa54-45dc-826d-c68d83d19eba" containerID="29be0999b77d1b7e8aa0983860a1f96dd2b656de5be9bee48fa0afb59329eec9" exitCode=0
Nov 02 17:45:50.010504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.010396    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq" event={"ID":"2805413d-aa54-45dc-826d-c68d83d19eba","Type":"ContainerDied","Data":"29be0999b77d1b7e8aa0983860a1f96dd2b656de5be9bee48fa0afb59329eec9"}
Nov 02 17:45:50.016678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.016640    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28" event={"ID":"e6db4524-c25c-4d26-9e3a-360cf0d72632","Type":"ContainerStarted","Data":"b4a6ca67d24954686d027fcc4afe42a1064592ad847ba8e0d7ad6ad2498de863"}
Nov 02 17:45:50.016810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.016703    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28" event={"ID":"e6db4524-c25c-4d26-9e3a-360cf0d72632","Type":"ContainerStarted","Data":"e6aff54520081f75e7f021825d33d9c8a85588daa40958501915c93040614521"}
Nov 02 17:45:50.040437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.040384    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28" podStartSLOduration=6.040368169 podStartE2EDuration="6.040368169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:50.036613495 +0000 UTC m=+762.616022883" watchObservedRunningTime="2026-11-02 17:45:50.040368169 +0000 UTC m=+762.619777555"
Nov 02 17:45:50.359274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.359233    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:50.429819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.405330    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca\") pod \"2805413d-aa54-45dc-826d-c68d83d19eba\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") "
Nov 02 17:45:50.429819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.405386    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9\" (UniqueName: \"kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9\") pod \"2805413d-aa54-45dc-826d-c68d83d19eba\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") "
Nov 02 17:45:50.429819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.405601    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca\") pod \"2805413d-aa54-45dc-826d-c68d83d19eba\" (UID: \"2805413d-aa54-45dc-826d-c68d83d19eba\") "
Nov 02 17:45:50.429819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.418937    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca" pod "2805413d-aa54-45dc-826d-c68d83d19eba" (UID: "2805413d-aa54-45dc-826d-c68d83d19eba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:50.429819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.429333    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca" pod "2805413d-aa54-45dc-826d-c68d83d19eba" (UID: "2805413d-aa54-45dc-826d-c68d83d19eba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:50.431772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.430974    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9" pod "2805413d-aa54-45dc-826d-c68d83d19eba" (UID: "2805413d-aa54-45dc-826d-c68d83d19eba"). InnerVolumeSpecName "kube-api-access-pn2c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:50.508864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.508067    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:50.508864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.508108    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2805413d-aa54-45dc-826d-c68d83d19eba-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:50.508864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.508123    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2c9\" (UniqueName: \"kubernetes.io/projected/2805413d-aa54-45dc-826d-c68d83d19eba-kube-api-access-pn2c9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:50.796718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.796686    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:50.797671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.797597    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:50.798696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:50.798675    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:45:51.028997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:51.028956    5042 generic.go:356] "Generic (PLEG): container finished" podID="e6db4524-c25c-4d26-9e3a-360cf0d72632" containerID="b4a6ca67d24954686d027fcc4afe42a1064592ad847ba8e0d7ad6ad2498de863" exitCode=0
Nov 02 17:45:51.029255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:51.029221    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28" event={"ID":"e6db4524-c25c-4d26-9e3a-360cf0d72632","Type":"ContainerDied","Data":"b4a6ca67d24954686d027fcc4afe42a1064592ad847ba8e0d7ad6ad2498de863"}
Nov 02 17:45:51.032221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:51.032197    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"
Nov 02 17:45:51.032992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:51.032961    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq" event={"ID":"2805413d-aa54-45dc-826d-c68d83d19eba","Type":"ContainerDied","Data":"48e7ee7e12bfc202ce3a9408b92ede127f181a6bbcce27e0194d82c84aea4f4e"}
Nov 02 17:45:51.033099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:51.033021    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e7ee7e12bfc202ce3a9408b92ede127f181a6bbcce27e0194d82c84aea4f4e"
Nov 02 17:45:52.506176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.506123    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:52.707569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.707469    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca\") pod \"e6db4524-c25c-4d26-9e3a-360cf0d72632\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") "
Nov 02 17:45:52.708108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.708085    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca\") pod \"e6db4524-c25c-4d26-9e3a-360cf0d72632\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") "
Nov 02 17:45:52.708259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.708240    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k\" (UniqueName: \"kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k\") pod \"e6db4524-c25c-4d26-9e3a-360cf0d72632\" (UID: \"e6db4524-c25c-4d26-9e3a-360cf0d72632\") "
Nov 02 17:45:52.709199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.709160    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca" pod "e6db4524-c25c-4d26-9e3a-360cf0d72632" (UID: "e6db4524-c25c-4d26-9e3a-360cf0d72632"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:52.713740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.712230    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k" pod "e6db4524-c25c-4d26-9e3a-360cf0d72632" (UID: "e6db4524-c25c-4d26-9e3a-360cf0d72632"). InnerVolumeSpecName "kube-api-access-s2w9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:52.714103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.714082    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:52.714251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.714236    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2w9k\" (UniqueName: \"kubernetes.io/projected/e6db4524-c25c-4d26-9e3a-360cf0d72632-kube-api-access-s2w9k\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:52.753877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.752255    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca" pod "e6db4524-c25c-4d26-9e3a-360cf0d72632" (UID: "e6db4524-c25c-4d26-9e3a-360cf0d72632"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:52.836866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:52.817941    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6db4524-c25c-4d26-9e3a-360cf0d72632-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:45:53.121915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:53.119106    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28" event={"ID":"e6db4524-c25c-4d26-9e3a-360cf0d72632","Type":"ContainerDied","Data":"e6aff54520081f75e7f021825d33d9c8a85588daa40958501915c93040614521"}
Nov 02 17:45:53.121915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:53.119175    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6aff54520081f75e7f021825d33d9c8a85588daa40958501915c93040614521"
Nov 02 17:45:53.121915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:53.119245    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"
Nov 02 17:45:54.187111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:54.185934    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:45:54.287865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:54.284833    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:54.309865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:54.308949    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:54.685758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:54.685033    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:45:54.837695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:54.837654    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:56.662825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.662786    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"]
Nov 02 17:45:56.663356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.663342    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2805413d-aa54-45dc-826d-c68d83d19eba" containerName="revoke-keys"
Nov 02 17:45:56.663399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.663356    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2805413d-aa54-45dc-826d-c68d83d19eba" containerName="revoke-keys"
Nov 02 17:45:56.663399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.663381    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e6db4524-c25c-4d26-9e3a-360cf0d72632" containerName="revoke-keys"
Nov 02 17:45:56.663399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.663387    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6db4524-c25c-4d26-9e3a-360cf0d72632" containerName="revoke-keys"
Nov 02 17:45:56.663488 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.663479    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6db4524-c25c-4d26-9e3a-360cf0d72632" containerName="revoke-keys"
Nov 02 17:45:56.663521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.663495    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2805413d-aa54-45dc-826d-c68d83d19eba" containerName="revoke-keys"
Nov 02 17:45:56.667163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.667145    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.673907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.673883    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:56.674030 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.673970    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:45:56.674109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.674049    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:45:56.678048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.678023    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:45:56.679566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.679541    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"]
Nov 02 17:45:56.821192 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.821159    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.821192 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.821193    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95hg\" (UniqueName: \"kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.821458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.821431    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.922495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.922396    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.922676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.922574    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.922676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.922600    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95hg\" (UniqueName: \"kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.923304 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.923225    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.925018 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.924994    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.931759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.931728    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95hg\" (UniqueName: \"kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg\") pod \"maas-api-revoke-sub-3da22a0b-rs7qr\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:56.990556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:56.990525    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:45:57.291936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:57.291889    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"]
Nov 02 17:45:57.293433 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:45:57.293394    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e501d6_d895_471e_aa91_a4c3bcad5e76.slice/crio-2db5759249a9cc51f4acea4d14484d2fdf4a6f1620438dbd0c50194a1ee2e07a WatchSource:0}: Error finding container 2db5759249a9cc51f4acea4d14484d2fdf4a6f1620438dbd0c50194a1ee2e07a: Status 404 returned error can't find the container with id 2db5759249a9cc51f4acea4d14484d2fdf4a6f1620438dbd0c50194a1ee2e07a
Nov 02 17:45:58.223676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:58.223641    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr" event={"ID":"54e501d6-d895-471e-aa91-a4c3bcad5e76","Type":"ContainerStarted","Data":"3cd41ca23cc563343ec1d099276b6df29edf5a2ebc31f4027ae38970bf8263ed"}
Nov 02 17:45:58.224154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:58.223692    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr" event={"ID":"54e501d6-d895-471e-aa91-a4c3bcad5e76","Type":"ContainerStarted","Data":"2db5759249a9cc51f4acea4d14484d2fdf4a6f1620438dbd0c50194a1ee2e07a"}
Nov 02 17:45:58.244051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:58.244003    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr" podStartSLOduration=2.243987542 podStartE2EDuration="2.243987542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:58.241113796 +0000 UTC m=+770.820523183" watchObservedRunningTime="2026-11-02 17:45:58.243987542 +0000 UTC m=+770.823396949"
Nov 02 17:45:59.233256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:59.233222    5042 generic.go:356] "Generic (PLEG): container finished" podID="54e501d6-d895-471e-aa91-a4c3bcad5e76" containerID="3cd41ca23cc563343ec1d099276b6df29edf5a2ebc31f4027ae38970bf8263ed" exitCode=0
Nov 02 17:45:59.233658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:45:59.233306    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr" event={"ID":"54e501d6-d895-471e-aa91-a4c3bcad5e76","Type":"ContainerDied","Data":"3cd41ca23cc563343ec1d099276b6df29edf5a2ebc31f4027ae38970bf8263ed"}
Nov 02 17:46:00.026326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.026291    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"]
Nov 02 17:46:00.030502 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.030483    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.041333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.041307    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"]
Nov 02 17:46:00.175394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.175357    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.175544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.175432    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.175592 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.175569    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjlb\" (UniqueName: \"kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.276548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.276454    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.277002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.276560    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.277002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.276609    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjlb\" (UniqueName: \"kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.277141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.277121    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.279175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.279145    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.285585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.285558    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjlb\" (UniqueName: \"kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb\") pod \"maas-api-revoke-sub-f6c47015-4szjx\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.346943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.346138    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:00.450302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.448070    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:46:00.582343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.581324    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca\") pod \"54e501d6-d895-471e-aa91-a4c3bcad5e76\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") "
Nov 02 17:46:00.582343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.581464    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca\") pod \"54e501d6-d895-471e-aa91-a4c3bcad5e76\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") "
Nov 02 17:46:00.582343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.581578    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg\" (UniqueName: \"kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg\") pod \"54e501d6-d895-471e-aa91-a4c3bcad5e76\" (UID: \"54e501d6-d895-471e-aa91-a4c3bcad5e76\") "
Nov 02 17:46:00.602872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.582663    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca" pod "54e501d6-d895-471e-aa91-a4c3bcad5e76" (UID: "54e501d6-d895-471e-aa91-a4c3bcad5e76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:00.602872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.587399    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg" pod "54e501d6-d895-471e-aa91-a4c3bcad5e76" (UID: "54e501d6-d895-471e-aa91-a4c3bcad5e76"). InnerVolumeSpecName "kube-api-access-c95hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:00.602872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.588282    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca" pod "54e501d6-d895-471e-aa91-a4c3bcad5e76" (UID: "54e501d6-d895-471e-aa91-a4c3bcad5e76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:00.657884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.657400    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"]
Nov 02 17:46:00.657884 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:00.657546    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0adafecb_5cdd_4a57_9230_80a649be16a3.slice/crio-91595b8945519586f8c68576b6f65e31988522412301d43f52ae0f0e96a189cd WatchSource:0}: Error finding container 91595b8945519586f8c68576b6f65e31988522412301d43f52ae0f0e96a189cd: Status 404 returned error can't find the container with id 91595b8945519586f8c68576b6f65e31988522412301d43f52ae0f0e96a189cd
Nov 02 17:46:00.685806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.684480    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:00.685806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.684520    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c95hg\" (UniqueName: \"kubernetes.io/projected/54e501d6-d895-471e-aa91-a4c3bcad5e76-kube-api-access-c95hg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:00.685806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:00.684536    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e501d6-d895-471e-aa91-a4c3bcad5e76-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:01.297911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:01.294207    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr" event={"ID":"54e501d6-d895-471e-aa91-a4c3bcad5e76","Type":"ContainerDied","Data":"2db5759249a9cc51f4acea4d14484d2fdf4a6f1620438dbd0c50194a1ee2e07a"}
Nov 02 17:46:01.297911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:01.294271    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db5759249a9cc51f4acea4d14484d2fdf4a6f1620438dbd0c50194a1ee2e07a"
Nov 02 17:46:01.297911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:01.294350    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"
Nov 02 17:46:01.323868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:01.315355    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx" event={"ID":"0adafecb-5cdd-4a57-9230-80a649be16a3","Type":"ContainerStarted","Data":"91595b8945519586f8c68576b6f65e31988522412301d43f52ae0f0e96a189cd"}
Nov 02 17:46:02.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:02.371919    5042 generic.go:356] "Generic (PLEG): container finished" podID="0adafecb-5cdd-4a57-9230-80a649be16a3" containerID="1f159dd9ab9896a01115ad823eb887b6414fa54ba57232e704f8110a82462ca2" exitCode=0
Nov 02 17:46:02.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:02.372067    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx" event={"ID":"0adafecb-5cdd-4a57-9230-80a649be16a3","Type":"ContainerDied","Data":"1f159dd9ab9896a01115ad823eb887b6414fa54ba57232e704f8110a82462ca2"}
Nov 02 17:46:03.570536 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.570511    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:03.657182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.657084    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb\" (UniqueName: \"kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb\") pod \"0adafecb-5cdd-4a57-9230-80a649be16a3\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") "
Nov 02 17:46:03.657374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.657191    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca\") pod \"0adafecb-5cdd-4a57-9230-80a649be16a3\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") "
Nov 02 17:46:03.657374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.657311    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca\") pod \"0adafecb-5cdd-4a57-9230-80a649be16a3\" (UID: \"0adafecb-5cdd-4a57-9230-80a649be16a3\") "
Nov 02 17:46:03.657739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.657707    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca" pod "0adafecb-5cdd-4a57-9230-80a649be16a3" (UID: "0adafecb-5cdd-4a57-9230-80a649be16a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:03.658295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.658271    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:03.659921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.659832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb" pod "0adafecb-5cdd-4a57-9230-80a649be16a3" (UID: "0adafecb-5cdd-4a57-9230-80a649be16a3"). InnerVolumeSpecName "kube-api-access-fhjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:03.681018 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.680972    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca" pod "0adafecb-5cdd-4a57-9230-80a649be16a3" (UID: "0adafecb-5cdd-4a57-9230-80a649be16a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:03.759562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.759533    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjlb\" (UniqueName: \"kubernetes.io/projected/0adafecb-5cdd-4a57-9230-80a649be16a3-kube-api-access-fhjlb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:03.759562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:03.759561    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0adafecb-5cdd-4a57-9230-80a649be16a3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:04.396833 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.396804    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"
Nov 02 17:46:04.396833 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.396829    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx" event={"ID":"0adafecb-5cdd-4a57-9230-80a649be16a3","Type":"ContainerDied","Data":"91595b8945519586f8c68576b6f65e31988522412301d43f52ae0f0e96a189cd"}
Nov 02 17:46:04.397063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.396904    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91595b8945519586f8c68576b6f65e31988522412301d43f52ae0f0e96a189cd"
Nov 02 17:46:04.473381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.473340    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"]
Nov 02 17:46:04.474216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.474194    5042 cpu_manager.go:414] "containerMap: removing container" podUID="54e501d6-d895-471e-aa91-a4c3bcad5e76" containerName="revoke-keys"
Nov 02 17:46:04.474216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.474214    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e501d6-d895-471e-aa91-a4c3bcad5e76" containerName="revoke-keys"
Nov 02 17:46:04.474404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.474237    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0adafecb-5cdd-4a57-9230-80a649be16a3" containerName="revoke-keys"
Nov 02 17:46:04.474404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.474244    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0adafecb-5cdd-4a57-9230-80a649be16a3" containerName="revoke-keys"
Nov 02 17:46:04.474701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.474674    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e501d6-d895-471e-aa91-a4c3bcad5e76" containerName="revoke-keys"
Nov 02 17:46:04.474767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.474702    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0adafecb-5cdd-4a57-9230-80a649be16a3" containerName="revoke-keys"
Nov 02 17:46:04.478973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.478952    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.484029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.483991    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:46:04.484029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.484012    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:46:04.484212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.484077    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:46:04.484212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.484085    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:46:04.487675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.487648    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"]
Nov 02 17:46:04.569391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.569356    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl9k8\" (UniqueName: \"kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.569553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.569527    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.569590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.569567    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.670506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.670411    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9k8\" (UniqueName: \"kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.671008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.670592    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.671008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.670656    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.671404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.671377    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.673295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.673270    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.679387 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.679361    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9k8\" (UniqueName: \"kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8\") pod \"maas-api-revoke-sub-ab722c4e-cwnnt\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:04.801130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:04.801095    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:05.090635 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:05.090591    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca8fa5d_f60c_4367_8d20_7c8967d941b0.slice/crio-cf102c2c69f93de8068ebe53ee17c06c878bce6b66c33eef027923f149e6fc5d WatchSource:0}: Error finding container cf102c2c69f93de8068ebe53ee17c06c878bce6b66c33eef027923f149e6fc5d: Status 404 returned error can't find the container with id cf102c2c69f93de8068ebe53ee17c06c878bce6b66c33eef027923f149e6fc5d
Nov 02 17:46:05.091518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:05.091485    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"]
Nov 02 17:46:05.405813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:05.405778    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt" event={"ID":"6ca8fa5d-f60c-4367-8d20-7c8967d941b0","Type":"ContainerStarted","Data":"2acc4285daac96a7cdf4e38e9b3c29f910797d43b8851673f74fe47721a4f696"}
Nov 02 17:46:05.406029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:05.405851    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt" event={"ID":"6ca8fa5d-f60c-4367-8d20-7c8967d941b0","Type":"ContainerStarted","Data":"cf102c2c69f93de8068ebe53ee17c06c878bce6b66c33eef027923f149e6fc5d"}
Nov 02 17:46:05.441813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:05.428643    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt" podStartSLOduration=1.428621607 podStartE2EDuration="1.428621607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:05.42158075 +0000 UTC m=+778.000990130" watchObservedRunningTime="2026-11-02 17:46:05.428621607 +0000 UTC m=+778.008030994"
Nov 02 17:46:06.415470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:06.415429    5042 generic.go:356] "Generic (PLEG): container finished" podID="6ca8fa5d-f60c-4367-8d20-7c8967d941b0" containerID="2acc4285daac96a7cdf4e38e9b3c29f910797d43b8851673f74fe47721a4f696" exitCode=0
Nov 02 17:46:06.415935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:06.415513    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt" event={"ID":"6ca8fa5d-f60c-4367-8d20-7c8967d941b0","Type":"ContainerDied","Data":"2acc4285daac96a7cdf4e38e9b3c29f910797d43b8851673f74fe47721a4f696"}
Nov 02 17:46:06.866314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:06.866280    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"]
Nov 02 17:46:06.870671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:06.870651    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:06.879907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:06.879879    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"]
Nov 02 17:46:07.022508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.022470    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.022660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.022612    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2vb\" (UniqueName: \"kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.022751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.022737    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.124464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.124368    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.124665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.124495    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.124665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.124566    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2vb\" (UniqueName: \"kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.125194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.125170    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.127041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.127015    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.132855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.132811    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2vb\" (UniqueName: \"kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb\") pod \"maas-api-revoke-sub-e0ed3068-nvdt7\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.192201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.192167    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:07.491164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.491132    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"]
Nov 02 17:46:07.687097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.687012    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:07.834662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.834630    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8\" (UniqueName: \"kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8\") pod \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") "
Nov 02 17:46:07.834875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.834829    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca\") pod \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") "
Nov 02 17:46:07.835164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.835066    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca\") pod \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\" (UID: \"6ca8fa5d-f60c-4367-8d20-7c8967d941b0\") "
Nov 02 17:46:07.835992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.835826    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca" pod "6ca8fa5d-f60c-4367-8d20-7c8967d941b0" (UID: "6ca8fa5d-f60c-4367-8d20-7c8967d941b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:07.838258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.838062    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca" pod "6ca8fa5d-f60c-4367-8d20-7c8967d941b0" (UID: "6ca8fa5d-f60c-4367-8d20-7c8967d941b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:07.839416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.839387    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8" pod "6ca8fa5d-f60c-4367-8d20-7c8967d941b0" (UID: "6ca8fa5d-f60c-4367-8d20-7c8967d941b0"). InnerVolumeSpecName "kube-api-access-tl9k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:07.936929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.936894    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl9k8\" (UniqueName: \"kubernetes.io/projected/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-kube-api-access-tl9k8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:07.936929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.936922    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:07.936929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:07.936933    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ca8fa5d-f60c-4367-8d20-7c8967d941b0-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:08.444101 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:08.444071    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"
Nov 02 17:46:08.444101 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:08.444079    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt" event={"ID":"6ca8fa5d-f60c-4367-8d20-7c8967d941b0","Type":"ContainerDied","Data":"cf102c2c69f93de8068ebe53ee17c06c878bce6b66c33eef027923f149e6fc5d"}
Nov 02 17:46:08.444353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:08.444139    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf102c2c69f93de8068ebe53ee17c06c878bce6b66c33eef027923f149e6fc5d"
Nov 02 17:46:08.445948 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:08.445923    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7" event={"ID":"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4","Type":"ContainerStarted","Data":"69de85df32fca7d9af21bffd57505e7b1b4d25e88a177154ad8e8991c8b891ba"}
Nov 02 17:46:08.446107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:08.445966    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7" event={"ID":"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4","Type":"ContainerStarted","Data":"36eb30d25cab8b1bc4f43ce226178db895f8a1007a384822f1cc7e1eb4bddf92"}
Nov 02 17:46:08.464865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:08.464756    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7" podStartSLOduration=2.46474163 podStartE2EDuration="2.46474163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:08.462921325 +0000 UTC m=+781.042330708" watchObservedRunningTime="2026-11-02 17:46:08.46474163 +0000 UTC m=+781.044151077"
Nov 02 17:46:09.455162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:09.455125    5042 generic.go:356] "Generic (PLEG): container finished" podID="7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" containerID="69de85df32fca7d9af21bffd57505e7b1b4d25e88a177154ad8e8991c8b891ba" exitCode=0
Nov 02 17:46:09.455579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:09.455175    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7" event={"ID":"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4","Type":"ContainerDied","Data":"69de85df32fca7d9af21bffd57505e7b1b4d25e88a177154ad8e8991c8b891ba"}
Nov 02 17:46:10.670715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.670688    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:10.785572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.785131    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca\") pod \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") "
Nov 02 17:46:10.788945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.788916    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb\" (UniqueName: \"kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb\") pod \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") "
Nov 02 17:46:10.789238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.789219    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca\") pod \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\" (UID: \"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4\") "
Nov 02 17:46:10.794915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.794785    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca" pod "7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" (UID: "7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:10.795505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.795453    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca" pod "7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" (UID: "7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:10.797899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.797832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb" pod "7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" (UID: "7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4"). InnerVolumeSpecName "kube-api-access-pv2vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:10.799459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.799387    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2vb\" (UniqueName: \"kubernetes.io/projected/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-kube-api-access-pv2vb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:10.799459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.799417    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:10.799459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:10.799436    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:11.658863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:11.599789    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"
Nov 02 17:46:11.676863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:11.670924    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7" event={"ID":"7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4","Type":"ContainerDied","Data":"36eb30d25cab8b1bc4f43ce226178db895f8a1007a384822f1cc7e1eb4bddf92"}
Nov 02 17:46:11.677509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:11.677483    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36eb30d25cab8b1bc4f43ce226178db895f8a1007a384822f1cc7e1eb4bddf92"
Nov 02 17:46:12.681911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:12.681833    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:46:12.835869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:12.835294    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:12.911867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:12.910938    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.090867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.089938    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.196892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.187951    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.300103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.300049    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.316011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.315959    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.683117 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.683075    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.684967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.684946    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:46:13.919725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.917962    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.919725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.918562    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:13.919725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.919699    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:46:13.920070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:13.919755    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:46:14.171017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.170966    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:14.171580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.171559    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:46:14.301863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.300943    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:14.301863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.301733    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:46:14.382865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.361956    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:14.398184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.392965    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:46:14.874831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.874790    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:14.960948 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.945200    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:46:14.961982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:14.961941    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:46:15.025496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.004919    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:46:15.187826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.186987    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:46:15.187826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.187466    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:15.387269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.387218    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:46:15.413554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.413476    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"]
Nov 02 17:46:15.414480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.414337    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" containerName="revoke-keys"
Nov 02 17:46:15.414480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.414360    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" containerName="revoke-keys"
Nov 02 17:46:15.414480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.414426    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6ca8fa5d-f60c-4367-8d20-7c8967d941b0" containerName="revoke-keys"
Nov 02 17:46:15.414480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.414434    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca8fa5d-f60c-4367-8d20-7c8967d941b0" containerName="revoke-keys"
Nov 02 17:46:15.414786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.414607    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca8fa5d-f60c-4367-8d20-7c8967d941b0" containerName="revoke-keys"
Nov 02 17:46:15.414786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.414624    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" containerName="revoke-keys"
Nov 02 17:46:15.511222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.511140    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.522195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.522163    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:46:15.569797 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.563236    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:46:15.583301 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.571522    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:46:15.583433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.573562    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:46:15.632868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.624949    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.632868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.625098    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.632868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.625152    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sb2w\" (UniqueName: \"kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.737864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.736755    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.737864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.736944    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.737864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.736996    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sb2w\" (UniqueName: \"kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.739985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.738957    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.822808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.822734    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.853112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.853079    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"]
Nov 02 17:46:15.897781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.895885    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:15.936863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.936387    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sb2w\" (UniqueName: \"kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w\") pod \"maas-api-revoke-sub-984f0fdd-6tkg5\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.960358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.960326    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:15.975145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.965386    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:15.989240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:15.985930    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:16.011399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:16.011355    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:16.093777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:16.093703    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:46:16.098485 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:16.098356    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:46:16.143512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:16.143483    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:46:16.164294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:16.164271    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:46:16.433919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:16.431408    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:46:17.091772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.091734    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"]
Nov 02 17:46:17.105060 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:17.105022    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acf1cd0_b383_433c_9db9_29accdd26a89.slice/crio-6ce51ef132a8a002c8902e27aaf9bb08027a0cf7d2b172be7bb22232c4fc72ed WatchSource:0}: Error finding container 6ce51ef132a8a002c8902e27aaf9bb08027a0cf7d2b172be7bb22232c4fc72ed: Status 404 returned error can't find the container with id 6ce51ef132a8a002c8902e27aaf9bb08027a0cf7d2b172be7bb22232c4fc72ed
Nov 02 17:46:17.432255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.432219    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"]
Nov 02 17:46:17.437784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.437760    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.447955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.447927    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"]
Nov 02 17:46:17.578659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.577891    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.578659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.578000    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9km\" (UniqueName: \"kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.578659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.578294    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.679693 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.679591    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.679895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.679720    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9km\" (UniqueName: \"kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.680428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.679974    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.680800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.680774    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.682251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.682232    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.691246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.691215    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9km\" (UniqueName: \"kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km\") pod \"maas-api-revoke-sub-2502ca91-sd7hn\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:17.758514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:17.758474    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:18.026153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:18.026095    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5" event={"ID":"3acf1cd0-b383-433c-9db9-29accdd26a89","Type":"ContainerStarted","Data":"eff2843d215b8612ed7a12d758cd75f9a025f45e8accabd2adff27251e15388c"}
Nov 02 17:46:18.026342 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:18.026170    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5" event={"ID":"3acf1cd0-b383-433c-9db9-29accdd26a89","Type":"ContainerStarted","Data":"6ce51ef132a8a002c8902e27aaf9bb08027a0cf7d2b172be7bb22232c4fc72ed"}
Nov 02 17:46:18.052684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:18.052640    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"]
Nov 02 17:46:18.059096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:18.059042    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5" podStartSLOduration=4.059023307 podStartE2EDuration="4.059023307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:18.048617175 +0000 UTC m=+790.628026556" watchObservedRunningTime="2026-11-02 17:46:18.059023307 +0000 UTC m=+790.638432713"
Nov 02 17:46:18.546632 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:46:18.546552    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:46:19.076866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:19.075975    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn" event={"ID":"a9250aec-db4b-4371-b805-a6f41d4c80f8","Type":"ContainerStarted","Data":"8c51716c83605a78bcc885620b868a9154078271825ddcd7d5339661dfdea6b7"}
Nov 02 17:46:19.076866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:19.076040    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn" event={"ID":"a9250aec-db4b-4371-b805-a6f41d4c80f8","Type":"ContainerStarted","Data":"04211cd8c342c812ee54a36d323520162372347c47c68d33a90aa28b9cc2fcb7"}
Nov 02 17:46:19.092865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:19.079481    5042 generic.go:356] "Generic (PLEG): container finished" podID="3acf1cd0-b383-433c-9db9-29accdd26a89" containerID="eff2843d215b8612ed7a12d758cd75f9a025f45e8accabd2adff27251e15388c" exitCode=0
Nov 02 17:46:19.092865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:19.079667    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5" event={"ID":"3acf1cd0-b383-433c-9db9-29accdd26a89","Type":"ContainerDied","Data":"eff2843d215b8612ed7a12d758cd75f9a025f45e8accabd2adff27251e15388c"}
Nov 02 17:46:19.140658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:19.140599    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn" podStartSLOduration=2.140579805 podStartE2EDuration="2.140579805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:19.114287792 +0000 UTC m=+791.693697179" watchObservedRunningTime="2026-11-02 17:46:19.140579805 +0000 UTC m=+791.719989192"
Nov 02 17:46:20.090063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.090029    5042 generic.go:356] "Generic (PLEG): container finished" podID="a9250aec-db4b-4371-b805-a6f41d4c80f8" containerID="8c51716c83605a78bcc885620b868a9154078271825ddcd7d5339661dfdea6b7" exitCode=0
Nov 02 17:46:20.090576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.090114    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn" event={"ID":"a9250aec-db4b-4371-b805-a6f41d4c80f8","Type":"ContainerDied","Data":"8c51716c83605a78bcc885620b868a9154078271825ddcd7d5339661dfdea6b7"}
Nov 02 17:46:20.269660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.269596    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:20.360130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.360047    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca\") pod \"3acf1cd0-b383-433c-9db9-29accdd26a89\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") "
Nov 02 17:46:20.360294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.360280    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca\") pod \"3acf1cd0-b383-433c-9db9-29accdd26a89\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") "
Nov 02 17:46:20.360382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.360363    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w\" (UniqueName: \"kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w\") pod \"3acf1cd0-b383-433c-9db9-29accdd26a89\" (UID: \"3acf1cd0-b383-433c-9db9-29accdd26a89\") "
Nov 02 17:46:20.360526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.360496    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca" pod "3acf1cd0-b383-433c-9db9-29accdd26a89" (UID: "3acf1cd0-b383-433c-9db9-29accdd26a89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:20.361001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.360918    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:20.362633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.362614    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w" pod "3acf1cd0-b383-433c-9db9-29accdd26a89" (UID: "3acf1cd0-b383-433c-9db9-29accdd26a89"). InnerVolumeSpecName "kube-api-access-7sb2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:20.362988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.362970    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca" pod "3acf1cd0-b383-433c-9db9-29accdd26a89" (UID: "3acf1cd0-b383-433c-9db9-29accdd26a89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:20.461981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.461946    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acf1cd0-b383-433c-9db9-29accdd26a89-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:20.461981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:20.461978    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sb2w\" (UniqueName: \"kubernetes.io/projected/3acf1cd0-b383-433c-9db9-29accdd26a89-kube-api-access-7sb2w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:21.098692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.098647    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5" event={"ID":"3acf1cd0-b383-433c-9db9-29accdd26a89","Type":"ContainerDied","Data":"6ce51ef132a8a002c8902e27aaf9bb08027a0cf7d2b172be7bb22232c4fc72ed"}
Nov 02 17:46:21.099246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.098733    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce51ef132a8a002c8902e27aaf9bb08027a0cf7d2b172be7bb22232c4fc72ed"
Nov 02 17:46:21.099246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.098663    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"
Nov 02 17:46:21.302681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.302648    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:21.373261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.373171    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca\") pod \"a9250aec-db4b-4371-b805-a6f41d4c80f8\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") "
Nov 02 17:46:21.373261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.373245    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca\") pod \"a9250aec-db4b-4371-b805-a6f41d4c80f8\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") "
Nov 02 17:46:21.373562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.373535    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca" pod "a9250aec-db4b-4371-b805-a6f41d4c80f8" (UID: "a9250aec-db4b-4371-b805-a6f41d4c80f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:21.373562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.373540    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km\" (UniqueName: \"kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km\") pod \"a9250aec-db4b-4371-b805-a6f41d4c80f8\" (UID: \"a9250aec-db4b-4371-b805-a6f41d4c80f8\") "
Nov 02 17:46:21.374121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.374099    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:21.376290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.376261    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca" pod "a9250aec-db4b-4371-b805-a6f41d4c80f8" (UID: "a9250aec-db4b-4371-b805-a6f41d4c80f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:21.376290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.376281    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km" pod "a9250aec-db4b-4371-b805-a6f41d4c80f8" (UID: "a9250aec-db4b-4371-b805-a6f41d4c80f8"). InnerVolumeSpecName "kube-api-access-xf9km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:21.475426 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.475381    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9km\" (UniqueName: \"kubernetes.io/projected/a9250aec-db4b-4371-b805-a6f41d4c80f8-kube-api-access-xf9km\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:21.475426 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:21.475419    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9250aec-db4b-4371-b805-a6f41d4c80f8-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:22.108433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:22.108403    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"
Nov 02 17:46:22.108860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:22.108433    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn" event={"ID":"a9250aec-db4b-4371-b805-a6f41d4c80f8","Type":"ContainerDied","Data":"04211cd8c342c812ee54a36d323520162372347c47c68d33a90aa28b9cc2fcb7"}
Nov 02 17:46:22.108860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:22.108496    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04211cd8c342c812ee54a36d323520162372347c47c68d33a90aa28b9cc2fcb7"
Nov 02 17:46:23.738814 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.738771    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"]
Nov 02 17:46:23.740402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.740344    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a9250aec-db4b-4371-b805-a6f41d4c80f8" containerName="revoke-keys"
Nov 02 17:46:23.741730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.741232    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9250aec-db4b-4371-b805-a6f41d4c80f8" containerName="revoke-keys"
Nov 02 17:46:23.741730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.741324    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3acf1cd0-b383-433c-9db9-29accdd26a89" containerName="revoke-keys"
Nov 02 17:46:23.741730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.741335    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acf1cd0-b383-433c-9db9-29accdd26a89" containerName="revoke-keys"
Nov 02 17:46:23.741730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.741579    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acf1cd0-b383-433c-9db9-29accdd26a89" containerName="revoke-keys"
Nov 02 17:46:23.741730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.741606    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9250aec-db4b-4371-b805-a6f41d4c80f8" containerName="revoke-keys"
Nov 02 17:46:23.746762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.746734    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"]
Nov 02 17:46:23.747161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.747145    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.756494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.756149    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:46:23.756494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.756192    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:46:23.756494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.756237    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:46:23.756494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.756261    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:46:23.812346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.812301    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68f7\" (UniqueName: \"kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.812543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.812472    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.812615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.812584    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.925331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.925294    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68f7\" (UniqueName: \"kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.925665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.925645    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.925936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.925917    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.927149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.927108    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.929279 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.929252    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:23.935218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:23.935193    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68f7\" (UniqueName: \"kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7\") pod \"maas-api-revoke-sub-79886bf2-bjm8l\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:24.084644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:24.084553    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:24.378259 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:24.378209    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d9105b_29aa_4587_87d2_72fcfe1dd012.slice/crio-4c10706da72e24d5a5ec900056c66a5ed9ea60d551efa9a87b53567b52cb94a0 WatchSource:0}: Error finding container 4c10706da72e24d5a5ec900056c66a5ed9ea60d551efa9a87b53567b52cb94a0: Status 404 returned error can't find the container with id 4c10706da72e24d5a5ec900056c66a5ed9ea60d551efa9a87b53567b52cb94a0
Nov 02 17:46:24.378395 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:24.378295    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"]
Nov 02 17:46:25.137229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:25.137129    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l" event={"ID":"42d9105b-29aa-4587-87d2-72fcfe1dd012","Type":"ContainerStarted","Data":"c3a9ff20473bc975bcc38993a6657164f52b980160e3111e66ac3b4b5d366992"}
Nov 02 17:46:25.137229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:25.137183    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l" event={"ID":"42d9105b-29aa-4587-87d2-72fcfe1dd012","Type":"ContainerStarted","Data":"4c10706da72e24d5a5ec900056c66a5ed9ea60d551efa9a87b53567b52cb94a0"}
Nov 02 17:46:25.156764 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:25.156706    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l" podStartSLOduration=2.156693152 podStartE2EDuration="2.156693152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:25.153393112 +0000 UTC m=+797.732802498" watchObservedRunningTime="2026-11-02 17:46:25.156693152 +0000 UTC m=+797.736102539"
Nov 02 17:46:26.149191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:26.149150    5042 generic.go:356] "Generic (PLEG): container finished" podID="42d9105b-29aa-4587-87d2-72fcfe1dd012" containerID="c3a9ff20473bc975bcc38993a6657164f52b980160e3111e66ac3b4b5d366992" exitCode=0
Nov 02 17:46:26.149660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:26.149258    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l" event={"ID":"42d9105b-29aa-4587-87d2-72fcfe1dd012","Type":"ContainerDied","Data":"c3a9ff20473bc975bcc38993a6657164f52b980160e3111e66ac3b4b5d366992"}
Nov 02 17:46:26.926889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:26.926302    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"]
Nov 02 17:46:26.947238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:26.947204    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"]
Nov 02 17:46:26.947465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:26.947280    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:26.955440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:26.955123    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2893d1-openshift-default-dockercfg-tgdcw\""
Nov 02 17:46:27.069230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069186    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2w2w\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069273    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069347    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069417    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069552 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069509    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069549    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069572    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069593    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.069711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.069668    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.170646    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.170720    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.170756    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.170787    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.170832    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171002    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2w2w\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171077    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171123    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171138    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171162    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171169    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.171375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171225    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.172633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.171915    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.173005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.172953    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.189439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.182146    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.189439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.184499    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.189439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.187488    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.189439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.188474    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2w2w\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w\") pod \"e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") " pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:27.369074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:27.315518    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:28.597921 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:46:28.597428    5042 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.021s"
Nov 02 17:46:28.597921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.597474    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"]
Nov 02 17:46:28.701075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.701036    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"]
Nov 02 17:46:28.728864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.702341    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.752877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.718522    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:28.762866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.746420    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l" event={"ID":"42d9105b-29aa-4587-87d2-72fcfe1dd012","Type":"ContainerDied","Data":"4c10706da72e24d5a5ec900056c66a5ed9ea60d551efa9a87b53567b52cb94a0"}
Nov 02 17:46:28.762866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.758277    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c10706da72e24d5a5ec900056c66a5ed9ea60d551efa9a87b53567b52cb94a0"
Nov 02 17:46:28.816886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.791733    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.820866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.817170    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-895q8\" (UniqueName: \"kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.883954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.861105    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.981151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.981114    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca\") pod \"42d9105b-29aa-4587-87d2-72fcfe1dd012\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") "
Nov 02 17:46:28.982821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.982795    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca\") pod \"42d9105b-29aa-4587-87d2-72fcfe1dd012\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") "
Nov 02 17:46:28.991612 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.991587    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7\" (UniqueName: \"kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7\") pod \"42d9105b-29aa-4587-87d2-72fcfe1dd012\" (UID: \"42d9105b-29aa-4587-87d2-72fcfe1dd012\") "
Nov 02 17:46:28.992013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.991069    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca" pod "42d9105b-29aa-4587-87d2-72fcfe1dd012" (UID: "42d9105b-29aa-4587-87d2-72fcfe1dd012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:28.996998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.996969    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.997180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.997162    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-895q8\" (UniqueName: \"kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.997438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.997418    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:28.997819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:28.997799    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:29.005898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.003673    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:29.153932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.134522    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-895q8\" (UniqueName: \"kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:29.153932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.134677    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7" pod "42d9105b-29aa-4587-87d2-72fcfe1dd012" (UID: "42d9105b-29aa-4587-87d2-72fcfe1dd012"). InnerVolumeSpecName "kube-api-access-l68f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:29.160010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.159970    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca" pod "42d9105b-29aa-4587-87d2-72fcfe1dd012" (UID: "42d9105b-29aa-4587-87d2-72fcfe1dd012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:29.190866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.190766    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01287c72-mz8b6\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:29.280895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.249582    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d9105b-29aa-4587-87d2-72fcfe1dd012-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:29.280895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.249617    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68f7\" (UniqueName: \"kubernetes.io/projected/42d9105b-29aa-4587-87d2-72fcfe1dd012-kube-api-access-l68f7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:29.401863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.401355    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:46:29.426364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.426207    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:46:29.451867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.427033    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:29.496876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.484957    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Nov 02 17:46:29.496876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.485491    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:46:29.496876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.485888    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:46:29.496876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.493813    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:46:29.812569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.494251    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:29.812569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.494284    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:46:29.862964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.494312    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Nov 02 17:46:30.140935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.539984    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:46:30.140935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.140271    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:30.140935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:29.651995    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:46:30.140935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.140725    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:30.177872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.174816    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:30.177872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.174945    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:30.177872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.174992    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:30.207757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.207703    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"]
Nov 02 17:46:30.403973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.403137    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:46:30.403973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.403576    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:46:30.440224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.440166    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:30.447872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.441494    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:46:30.512201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.499288    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:46:30.512201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.507247    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:46:30.512201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.508111    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:46:30.512201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.508174    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:46:30.658219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.658025    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 02 17:46:30.795866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.775743    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:30.922346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.847974    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:46:30.924695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.848015    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:46:31.144202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:30.922940    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:46:31.145911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.145888    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:46:31.146173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.146140    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:46:31.378388 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.378354    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" event={"ID":"dee90570-f1bb-40a4-b8fb-59c3f53c52dc","Type":"ContainerStarted","Data":"e959f931f0619b7718dc7254a374a5eee7b0f2843455e12215d0a8e0948462ef"}
Nov 02 17:46:31.411165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.411115    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:31.412654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.412627    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:46:31.412913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.412886    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:31.413360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.413342    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"
Nov 02 17:46:31.475868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.450677    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:31.529590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.499525    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:31.530953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.529096    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:31.583998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.583954    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:31.926752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.926077    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:46:31.935801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:31.927166    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:46:32.198393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.147965    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:32.198393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.160779    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-m6g9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:46:32.198393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.160874    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:32.219072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.219026    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:32.285811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.285564    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:32.434878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.433945    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:46:32.676153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.639210    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:32.676153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.639655    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:46:32.676153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.639991    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:46:32.988883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.987169    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:46:32.988883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:32.987954    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:46:33.270025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.269916    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:46:33.322665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.322621    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:46:33.445558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.445525    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:46:33.720248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.628784    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:46:33.720248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.636020    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:33.720248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.666503    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:33.720248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.699744    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:46:33.720248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.699816    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:33.839462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.819480    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:46:33.839462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:33.819987    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:46:34.049143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.049098    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:46:34.398188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.393619    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:34.446110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.446052    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:34.512862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.450541    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" event={"ID":"dee90570-f1bb-40a4-b8fb-59c3f53c52dc","Type":"ContainerStarted","Data":"52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"}
Nov 02 17:46:34.569644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.569523    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6" event={"ID":"6642c09e-eabf-4215-85ed-32fff0324e5d","Type":"ContainerStarted","Data":"76967a95e4b086d8de627bd9b95cffb855983587746b2bfdf22f843b261954d6"}
Nov 02 17:46:34.632358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.632272    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"]
Nov 02 17:46:34.716121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.715807    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:46:34.728866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.716596    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:34.730649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.730586    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podStartSLOduration=8.730563222 podStartE2EDuration="8.730563222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:34.562277339 +0000 UTC m=+807.141686725" watchObservedRunningTime="2026-11-02 17:46:34.730563222 +0000 UTC m=+807.309972609"
Nov 02 17:46:34.820800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.820757    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:46:34.821280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.821251    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:46:34.972877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.969268    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:46:34.976886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:34.973936    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:46:35.035736 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.027184    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:46:35.039882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.038628    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:46:35.316939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.316822    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:35.322682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.322652    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:35.580651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.579969    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6" event={"ID":"6642c09e-eabf-4215-85ed-32fff0324e5d","Type":"ContainerStarted","Data":"fb933a8799ee2a87f9e7662908b77a3f4c85485bc68c54cac94b0c7744420b73"}
Nov 02 17:46:35.580819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.580702    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:35.581818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.581798    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:46:35.602636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:35.602592    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6" podStartSLOduration=8.602579925 podStartE2EDuration="8.602579925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:35.599242598 +0000 UTC m=+808.178651981" watchObservedRunningTime="2026-11-02 17:46:35.602579925 +0000 UTC m=+808.181989310"
Nov 02 17:46:36.593463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:36.593427    5042 generic.go:356] "Generic (PLEG): container finished" podID="6642c09e-eabf-4215-85ed-32fff0324e5d" containerID="fb933a8799ee2a87f9e7662908b77a3f4c85485bc68c54cac94b0c7744420b73" exitCode=0
Nov 02 17:46:36.593936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:36.593522    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6" event={"ID":"6642c09e-eabf-4215-85ed-32fff0324e5d","Type":"ContainerDied","Data":"fb933a8799ee2a87f9e7662908b77a3f4c85485bc68c54cac94b0c7744420b73"}
Nov 02 17:46:37.823108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.823072    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:37.860466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.860438    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca\") pod \"6642c09e-eabf-4215-85ed-32fff0324e5d\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") "
Nov 02 17:46:37.860632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.860522    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8\" (UniqueName: \"kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8\") pod \"6642c09e-eabf-4215-85ed-32fff0324e5d\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") "
Nov 02 17:46:37.860632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.860604    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca\") pod \"6642c09e-eabf-4215-85ed-32fff0324e5d\" (UID: \"6642c09e-eabf-4215-85ed-32fff0324e5d\") "
Nov 02 17:46:37.860932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.860907    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca" pod "6642c09e-eabf-4215-85ed-32fff0324e5d" (UID: "6642c09e-eabf-4215-85ed-32fff0324e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:37.861354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.861326    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:37.863267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.863212    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca" pod "6642c09e-eabf-4215-85ed-32fff0324e5d" (UID: "6642c09e-eabf-4215-85ed-32fff0324e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:37.863267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.863242    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8" pod "6642c09e-eabf-4215-85ed-32fff0324e5d" (UID: "6642c09e-eabf-4215-85ed-32fff0324e5d"). InnerVolumeSpecName "kube-api-access-895q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:37.963202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.963169    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6642c09e-eabf-4215-85ed-32fff0324e5d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:37.963202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:37.963198    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-895q8\" (UniqueName: \"kubernetes.io/projected/6642c09e-eabf-4215-85ed-32fff0324e5d-kube-api-access-895q8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:46:38.617597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:38.617558    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6" event={"ID":"6642c09e-eabf-4215-85ed-32fff0324e5d","Type":"ContainerDied","Data":"76967a95e4b086d8de627bd9b95cffb855983587746b2bfdf22f843b261954d6"}
Nov 02 17:46:38.617876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:38.617621    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76967a95e4b086d8de627bd9b95cffb855983587746b2bfdf22f843b261954d6"
Nov 02 17:46:38.617876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:38.617577    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"
Nov 02 17:46:48.552567 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:46:48.552525    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:46:50.769915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.769875    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"]
Nov 02 17:46:50.770729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.770710    5042 cpu_manager.go:414] "containerMap: removing container" podUID="42d9105b-29aa-4587-87d2-72fcfe1dd012" containerName="revoke-keys"
Nov 02 17:46:50.770802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.770733    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d9105b-29aa-4587-87d2-72fcfe1dd012" containerName="revoke-keys"
Nov 02 17:46:50.770802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.770756    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6642c09e-eabf-4215-85ed-32fff0324e5d" containerName="revoke-keys"
Nov 02 17:46:50.770802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.770764    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6642c09e-eabf-4215-85ed-32fff0324e5d" containerName="revoke-keys"
Nov 02 17:46:50.771044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.770926    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d9105b-29aa-4587-87d2-72fcfe1dd012" containerName="revoke-keys"
Nov 02 17:46:50.771044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.770940    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6642c09e-eabf-4215-85ed-32fff0324e5d" containerName="revoke-keys"
Nov 02 17:46:50.776979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.776955    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.804133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.804080    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"]
Nov 02 17:46:50.805067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.805039    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-58970f-openshift-default-dockercfg-2qrss\""
Nov 02 17:46:50.962861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.939755    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.963157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.963129    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.963269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.963256    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.963410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.963398    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.963591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.963579    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.963744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.963731    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.974864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.967994    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.975166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.975139    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7zmx\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:50.975297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:50.975281    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076501    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076597    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076699    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076746    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7zmx\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076781    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076891    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.076934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076915    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.077183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.076944    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.077183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.077004    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.077594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.077559    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.077875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.077815    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.078106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.078071    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.078753 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.078732    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.079260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.079235    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.081465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.081443    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.081660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.081640    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.087125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.087098    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.087334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.087313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7zmx\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx\") pod \"e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") " pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.131283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.131240    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:51.447550 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:51.447519    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634bad02_9a5c_43a4_909b_a6d8b5e8c968.slice/crio-e28c621f8620ec3c57d5dd152cfdfd0ed4085677abf2b3dfd566aa0786cbffd3 WatchSource:0}: Error finding container e28c621f8620ec3c57d5dd152cfdfd0ed4085677abf2b3dfd566aa0786cbffd3: Status 404 returned error can't find the container with id e28c621f8620ec3c57d5dd152cfdfd0ed4085677abf2b3dfd566aa0786cbffd3
Nov 02 17:46:51.448541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.448503    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"]
Nov 02 17:46:51.450692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.450655    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:51.450799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.450726    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:51.450799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.450758    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:51.849961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.840544    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" event={"ID":"634bad02-9a5c-43a4-909b-a6d8b5e8c968","Type":"ContainerStarted","Data":"e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"}
Nov 02 17:46:51.849961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.840610    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" event={"ID":"634bad02-9a5c-43a4-909b-a6d8b5e8c968","Type":"ContainerStarted","Data":"e28c621f8620ec3c57d5dd152cfdfd0ed4085677abf2b3dfd566aa0786cbffd3"}
Nov 02 17:46:51.873899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:51.873817    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podStartSLOduration=1.873797798 podStartE2EDuration="1.873797798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:51.867338459 +0000 UTC m=+824.446747848" watchObservedRunningTime="2026-11-02 17:46:51.873797798 +0000 UTC m=+824.453207185"
Nov 02 17:46:52.132513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.132434    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:52.137572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.137551    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:52.353139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.353100    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"]
Nov 02 17:46:52.358365 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.358341    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.369702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.369671    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"]
Nov 02 17:46:52.396351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.396286    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"]
Nov 02 17:46:52.400518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.400496    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.401265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401228    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401291    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401370    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401424    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsg5v\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401488    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401557    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401657 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401641    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401695    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.401793 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.401778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.427892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.417252    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"]
Nov 02 17:46:52.503325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503291    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503330    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.503503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503355    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503438    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.503503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503465    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503505    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.503717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503552    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.503717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503681    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.503913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503744    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503762    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.503913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503811    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503834    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503866    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.503913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503902    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.504199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503929    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.504199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.503995    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsg5v\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.504199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.504045    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.504199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.504110    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.504199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.504134    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxh5t\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.504462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.504244    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.504663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.504642    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.505112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.504905    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.505112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.505082    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.506281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.506253    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.506614 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.506596    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.512499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.512477    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.512936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.512918    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsg5v\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.605882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.605808    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.605913    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxh5t\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606025    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606133    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606181    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606212    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606266    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606380    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606483 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606466    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606491    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.606734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.606709    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.607189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.607166    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.607327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.607276    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.607457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.607341    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.609438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.609413    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.609738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.609718    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.615000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.614974    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.615329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.615295    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxh5t\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.679492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.679402    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:52.731606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.731563    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:52.866753 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.866633    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:52.867744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:52.867723    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:46:53.044336 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:53.044293    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfe1f83_d393_4cd4_854d_07ab396dbb0e.slice/crio-ffc70da4594a9ba607516d0077cf3afc92305ebcf61ff259134a6ed4c7d8fae7 WatchSource:0}: Error finding container ffc70da4594a9ba607516d0077cf3afc92305ebcf61ff259134a6ed4c7d8fae7: Status 404 returned error can't find the container with id ffc70da4594a9ba607516d0077cf3afc92305ebcf61ff259134a6ed4c7d8fae7
Nov 02 17:46:53.047691 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.047650    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:53.047767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.047740    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:53.047811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.047780    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:53.047978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.047961    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"]
Nov 02 17:46:53.084298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.084268    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"]
Nov 02 17:46:53.085503 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:46:53.085470    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc235e191_4fc2_4e92_af3c_c2de4b59c9ff.slice/crio-3c46fcb6d705602fb66f6d4e5c732c616a84f87d6252993b5b89d14044e93a74 WatchSource:0}: Error finding container 3c46fcb6d705602fb66f6d4e5c732c616a84f87d6252993b5b89d14044e93a74: Status 404 returned error can't find the container with id 3c46fcb6d705602fb66f6d4e5c732c616a84f87d6252993b5b89d14044e93a74
Nov 02 17:46:53.088923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.088887    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:53.089033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.088970    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:53.089033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.089014    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:53.958626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.958585    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" event={"ID":"c235e191-4fc2-4e92-af3c-c2de4b59c9ff","Type":"ContainerStarted","Data":"ba116e2e8972a30dd1a1485552dbe39a88cbfc12a0d1d6db8ca36297bbd18429"}
Nov 02 17:46:53.959299 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.959274    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" event={"ID":"c235e191-4fc2-4e92-af3c-c2de4b59c9ff","Type":"ContainerStarted","Data":"3c46fcb6d705602fb66f6d4e5c732c616a84f87d6252993b5b89d14044e93a74"}
Nov 02 17:46:53.962826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.962801    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" event={"ID":"9cfe1f83-d393-4cd4-854d-07ab396dbb0e","Type":"ContainerStarted","Data":"0490fcbf60bd5581eea31e50d8ba056aea58afe06ce10062e31fe9ce44c5c6e0"}
Nov 02 17:46:53.963095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:53.963075    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" event={"ID":"9cfe1f83-d393-4cd4-854d-07ab396dbb0e","Type":"ContainerStarted","Data":"ffc70da4594a9ba607516d0077cf3afc92305ebcf61ff259134a6ed4c7d8fae7"}
Nov 02 17:46:54.039060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.038811    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podStartSLOduration=2.038795129 podStartE2EDuration="2.038795129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:54.038744825 +0000 UTC m=+826.618154211" watchObservedRunningTime="2026-11-02 17:46:54.038795129 +0000 UTC m=+826.618204516"
Nov 02 17:46:54.063473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.063419    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podStartSLOduration=2.063405295 podStartE2EDuration="2.063405295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:54.05955916 +0000 UTC m=+826.638968547" watchObservedRunningTime="2026-11-02 17:46:54.063405295 +0000 UTC m=+826.642814742"
Nov 02 17:46:54.737920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.736491    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:46:54.737920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.736715    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:46:54.754535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.751433    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:46:54.754535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.751488    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:46:54.754535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.751789    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:46:54.754535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:54.751821    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:46:55.680402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:55.680369    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:46:55.681020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:55.680988    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:46:55.736361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:55.736308    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:46:55.736673 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:55.736625    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:46:56.684867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:56.681331    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:46:56.684867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:56.681414    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:46:56.744034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:56.743994    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:46:56.744234 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:56.744071    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:46:57.682866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:57.680223    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:46:57.682866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:57.680324    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:46:57.734870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:57.733359    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:46:57.734870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:57.733447    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:46:58.689247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:58.689208    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:46:58.689547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:58.689519    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:46:58.767864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:58.767594    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:46:58.767864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:58.767688    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:46:59.683389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:59.683356    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:46:59.683722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:59.683696    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:46:59.733457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:59.733424    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:46:59.733723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:46:59.733695    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:47:00.680158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:00.680115    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body=
Nov 02 17:47:00.680667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:00.680216    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused"
Nov 02 17:47:00.732615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:00.732581    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 17:47:00.732931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:00.732636    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 17:47:00.995834 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:00.995744    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"]
Nov 02 17:47:01.008115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.008083    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.018000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.017966    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"]
Nov 02 17:47:01.165480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165437    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qkf8\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.165671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165587    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.165671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165631    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.165671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165659    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.165896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165694    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.165896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165723    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.165896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165810    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.166076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165911    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.166076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.165992    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.271292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.271174    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.272884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.272057    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.273063    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332086    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332261    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qkf8\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332358    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332395    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332431    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332476    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.332505    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.333131    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.333989    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.335473    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.348864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.336048    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.353444 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.353413    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.355925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.355303    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.382986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.381422    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.382986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.390052    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qkf8\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8\") pod \"e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") " pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.673991 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.673890    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:01.686961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.686923    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:47:01.688191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.688165    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:47:01.689310 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.689287    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:47:01.738071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.738029    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:47:01.739155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.739095    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:47:01.739979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:01.739956    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:47:02.068171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:02.068141    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"]
Nov 02 17:47:02.068448 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:47:02.068387    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b74354_9f55_4fc1_b42b_496a5838f504.slice/crio-c703090d7024a0c0a95b86846b47041db7feafc52955ec2928fd5474d1bcd693 WatchSource:0}: Error finding container c703090d7024a0c0a95b86846b47041db7feafc52955ec2928fd5474d1bcd693: Status 404 returned error can't find the container with id c703090d7024a0c0a95b86846b47041db7feafc52955ec2928fd5474d1bcd693
Nov 02 17:47:02.071568 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:02.071526    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:02.071734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:02.071611    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:02.071734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:02.071654    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:02.158113 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:02.158075    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" event={"ID":"21b74354-9f55-4fc1-b42b-496a5838f504","Type":"ContainerStarted","Data":"c703090d7024a0c0a95b86846b47041db7feafc52955ec2928fd5474d1bcd693"}
Nov 02 17:47:03.236685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:03.236657    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" event={"ID":"21b74354-9f55-4fc1-b42b-496a5838f504","Type":"ContainerStarted","Data":"292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6"}
Nov 02 17:47:03.288865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:03.272729    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podStartSLOduration=3.272709134 podStartE2EDuration="3.272709134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:03.267463317 +0000 UTC m=+835.846872704" watchObservedRunningTime="2026-11-02 17:47:03.272709134 +0000 UTC m=+835.852118523"
Nov 02 17:47:03.725167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:03.724179    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:03.725167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:03.724233    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:03.729050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:03.729030    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:04.677179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:04.677145    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:04.677794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:04.677764    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:05.679868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:05.679280    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:05.679868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:05.679357    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:06.974009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:06.972287    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:06.974009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:06.972353    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:07.919878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:07.911288    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:07.919878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:07.913593    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:07.955642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:07.938862    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:07.955642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:07.939349    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:47:08.157769 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.157727    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:47:08.267978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.260378    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"]
Nov 02 17:47:08.269297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.269052    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"]
Nov 02 17:47:08.269297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.269169    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.435552 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.435512    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.438697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.438664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.439099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439075    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.439315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439296    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.439484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439468    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.439615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439599    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.439739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439724    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbfmp\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.439872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439857    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.440002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.439987    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.542731    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.542821    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.542892    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.542933    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbfmp\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.542963    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.543000    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.543121    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.543212    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.543262    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.543806    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.544583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.544090    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.546893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.545799    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.547811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.547778    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.548205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.548183    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.548529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.548513    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.553779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.553415    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.556610 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.556584    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.557030 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.557012    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbfmp\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:08.690863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.690329    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:08.690863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.690394    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:08.743313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:08.743278    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:09.260213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.256789    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:09.260213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.256900    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:09.260213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.256948    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:09.264267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.263552    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"]
Nov 02 17:47:09.462883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.462833    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" event={"ID":"ff95a5d7-5e13-4987-9a9f-6c10ced46438","Type":"ContainerStarted","Data":"f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895"}
Nov 02 17:47:09.463082 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.462918    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" event={"ID":"ff95a5d7-5e13-4987-9a9f-6c10ced46438","Type":"ContainerStarted","Data":"83a0053a34c27d1ced15a65fac61d7eb7757411e6b7eacad5a77ea43471e0106"}
Nov 02 17:47:09.491975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.485328    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podStartSLOduration=2.485307496 podStartE2EDuration="2.485307496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:09.484281732 +0000 UTC m=+842.063691119" watchObservedRunningTime="2026-11-02 17:47:09.485307496 +0000 UTC m=+842.064716883"
Nov 02 17:47:09.702990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.702957    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 17:47:09.703256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.703228    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 17:47:09.743770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:09.743687    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:10.763860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:10.760972    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:10.763860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:10.761033    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:11.369025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:11.367630    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:11.369235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:11.369100    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:11.369934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:11.369913    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:47:11.795870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:11.745461    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:11.795870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:11.745534    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:12.743922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:12.743887    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:12.744093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:12.743939    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:13.745031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:13.744507    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:13.745031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:13.744568    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:14.744637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:14.744596    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:14.744929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:14.744664    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:15.329863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.328041    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"]
Nov 02 17:47:15.350618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.349382    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"]
Nov 02 17:47:15.350618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.349507    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.356283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.355869    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2129b9-openshift-default-dockercfg-jpd92\""
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.538742    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wztmc\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.538899    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.538940    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.539024    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.539055    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.539149    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.539264    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.539321    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.539625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.539411    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669031    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669080    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669145    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669171    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669239    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669324    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669376    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669457    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.669921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.669548    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wztmc\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.701873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.674514    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.701873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.676258    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.701873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.676517    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.701873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.677152    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.701873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.699141    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.768045 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.755085    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:15.768045 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.755161    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:15.848228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.848142    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wztmc\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.849170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.849139    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.853242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.853193    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:15.870440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:15.870404    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy\") pod \"e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") " pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:16.013386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.013303    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:16.344671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.344640    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"]
Nov 02 17:47:16.347117 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:47:16.347083    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a52f6d_c279_4d1a_b201_66d2e1e75109.slice/crio-452485934c8d7091a6ebcf8d63590d14d575128a74a939901220cb637c44b757 WatchSource:0}: Error finding container 452485934c8d7091a6ebcf8d63590d14d575128a74a939901220cb637c44b757: Status 404 returned error can't find the container with id 452485934c8d7091a6ebcf8d63590d14d575128a74a939901220cb637c44b757
Nov 02 17:47:16.350561 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.350521    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:16.350651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.350613    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:16.350691 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.350653    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:16.738229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.738191    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"]
Nov 02 17:47:16.749568 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.749531    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 02 17:47:16.749764 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.749598    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 02 17:47:16.752274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.752247    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.757904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.753000    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"]
Nov 02 17:47:16.759123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.759096    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:47:16.765134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.760227    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:47:16.765134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.760286    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:47:16.765134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.760349    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:47:16.802559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.802526    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.802765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.802588    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.802765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.802654    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tpb\" (UniqueName: \"kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.834051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.834004    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" event={"ID":"55a52f6d-c279-4d1a-b201-66d2e1e75109","Type":"ContainerStarted","Data":"d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf"}
Nov 02 17:47:16.834248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.834083    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" event={"ID":"55a52f6d-c279-4d1a-b201-66d2e1e75109","Type":"ContainerStarted","Data":"452485934c8d7091a6ebcf8d63590d14d575128a74a939901220cb637c44b757"}
Nov 02 17:47:16.879077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.879010    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" podStartSLOduration=1.878989202 podStartE2EDuration="1.878989202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:16.871155869 +0000 UTC m=+849.450565258" watchObservedRunningTime="2026-11-02 17:47:16.878989202 +0000 UTC m=+849.458398591"
Nov 02 17:47:16.904606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.904550    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.904810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.904648    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.904810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.904773    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tpb\" (UniqueName: \"kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.906164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.906132    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.910484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.910456    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:16.926967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:16.926504    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tpb\" (UniqueName: \"kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb\") pod \"maas-api-revoke-sub-9c2dba23-5k6mf\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:17.066378 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.066278    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:17.089901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.084595    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:17.089901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.086360    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:17.400040 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:47:17.400002    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fa5824_f698_4ddc_8f8e_5620d98b0c8d.slice/crio-88e981a08d6664668f9c99639230c201f0c73b975fe278fa23cf4c638d168008 WatchSource:0}: Error finding container 88e981a08d6664668f9c99639230c201f0c73b975fe278fa23cf4c638d168008: Status 404 returned error can't find the container with id 88e981a08d6664668f9c99639230c201f0c73b975fe278fa23cf4c638d168008
Nov 02 17:47:17.401461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.401421    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"]
Nov 02 17:47:17.846340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.846309    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:17.846913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.846878    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:17.848220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.848199    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 17:47:17.850572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.850544    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf" event={"ID":"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d","Type":"ContainerStarted","Data":"88e981a08d6664668f9c99639230c201f0c73b975fe278fa23cf4c638d168008"}
Nov 02 17:47:17.850964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.850945    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:17.852163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:17.852139    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:47:18.544389 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:47:18.544354    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:47:18.860061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.859969    5042 generic.go:356] "Generic (PLEG): container finished" podID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerID="52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259" exitCode=0
Nov 02 17:47:18.860237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.860050    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" event={"ID":"dee90570-f1bb-40a4-b8fb-59c3f53c52dc","Type":"ContainerDied","Data":"52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"}
Nov 02 17:47:18.860575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.860546    5042 scope.go:122] "RemoveContainer" containerID="52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"
Nov 02 17:47:18.862037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.862015    5042 generic.go:356] "Generic (PLEG): container finished" podID="f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" containerID="3df0cdee42d427ea3a19908d0d3e0e3fac430578ea8f1d3b473a2402d47c2c6c" exitCode=0
Nov 02 17:47:18.862134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.862044    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf" event={"ID":"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d","Type":"ContainerDied","Data":"3df0cdee42d427ea3a19908d0d3e0e3fac430578ea8f1d3b473a2402d47c2c6c"}
Nov 02 17:47:18.862239 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.862209    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:18.862295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.862282    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:18.862329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:18.862322    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:19.899864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:19.896181    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" event={"ID":"dee90570-f1bb-40a4-b8fb-59c3f53c52dc","Type":"ContainerStarted","Data":"593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa"}
Nov 02 17:47:20.157707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.157645    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:20.234447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.234409    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb\" (UniqueName: \"kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb\") pod \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") "
Nov 02 17:47:20.234649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.234467    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca\") pod \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") "
Nov 02 17:47:20.234649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.234538    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca\") pod \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\" (UID: \"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d\") "
Nov 02 17:47:20.235264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.235241    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca" pod "f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" (UID: "f7fa5824-f698-4ddc-8f8e-5620d98b0c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:20.237802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.237771    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca" pod "f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" (UID: "f7fa5824-f698-4ddc-8f8e-5620d98b0c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:47:20.238010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.237985    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb" pod "f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" (UID: "f7fa5824-f698-4ddc-8f8e-5620d98b0c8d"). InnerVolumeSpecName "kube-api-access-k5tpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:20.317695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.317510    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:47:20.318499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.318473    5042 patch_prober.go:29] interesting pod/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:47:20.318572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.318523    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 17:47:20.336094 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.336070    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tpb\" (UniqueName: \"kubernetes.io/projected/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-kube-api-access-k5tpb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:20.336094 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.336093    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:20.336235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.336103    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:20.597916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.585940    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:47:20.929190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.929094    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf" event={"ID":"f7fa5824-f698-4ddc-8f8e-5620d98b0c8d","Type":"ContainerDied","Data":"88e981a08d6664668f9c99639230c201f0c73b975fe278fa23cf4c638d168008"}
Nov 02 17:47:20.929190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.929128    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"
Nov 02 17:47:20.929190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:20.929162    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e981a08d6664668f9c99639230c201f0c73b975fe278fa23cf4c638d168008"
Nov 02 17:47:21.317376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:21.317312    5042 patch_prober.go:29] interesting pod/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:47:21.317376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:21.317364    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 17:47:22.322300 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:22.322242    5042 patch_prober.go:29] interesting pod/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:47:22.322965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:22.322932    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 17:47:23.317035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:23.316997    5042 patch_prober.go:29] interesting pod/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:47:23.317247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:23.317064    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 17:47:24.320722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:24.320249    5042 patch_prober.go:29] interesting pod/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:47:24.320722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:24.320345    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Nov 02 17:47:25.323080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:25.323045    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:47:25.324331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:25.324311    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:47:27.728618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.728577    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"]
Nov 02 17:47:27.729506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.729484    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" containerName="revoke-keys"
Nov 02 17:47:27.729506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.729504    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" containerName="revoke-keys"
Nov 02 17:47:27.729699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.729687    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" containerName="revoke-keys"
Nov 02 17:47:27.741946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.741922    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.746780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.746749    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"]
Nov 02 17:47:27.758408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.758386    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a976bc-openshift-default-dockercfg-qwjsm\""
Nov 02 17:47:27.880741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.880706    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.880956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.880747    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.880956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.880792    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.881083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.881039    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.881140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.881079    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.881313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.881293    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.881425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.881335    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxp5c\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.881500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.881478    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.881583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.881567    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.983799    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.983872    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984014    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984060    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxp5c\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984142    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984175    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984223    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984261    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984323    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.984859    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.985139    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:27.993917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.986181    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.015863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.998693    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.015863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:27.999001    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.015863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.001165    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.029864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.022059    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.042973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.040407    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.061888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.058949    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxp5c\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.096874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.088746    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:28.588341 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:47:28.588307    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf39e49_e7f0_4e3c_bbc7_781a41f12dda.slice/crio-b47ab5e6ed3e1560c9977f16b5a44f800faa9b0890fa0a06050e1450e78c061e WatchSource:0}: Error finding container b47ab5e6ed3e1560c9977f16b5a44f800faa9b0890fa0a06050e1450e78c061e: Status 404 returned error can't find the container with id b47ab5e6ed3e1560c9977f16b5a44f800faa9b0890fa0a06050e1450e78c061e
Nov 02 17:47:28.590377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.590347    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"]
Nov 02 17:47:28.591780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.591737    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:28.591920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.591827    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:28.591920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:28.591898    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:29.149912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:29.142383    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" event={"ID":"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda","Type":"ContainerStarted","Data":"3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d"}
Nov 02 17:47:29.149912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:29.142489    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" event={"ID":"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda","Type":"ContainerStarted","Data":"b47ab5e6ed3e1560c9977f16b5a44f800faa9b0890fa0a06050e1450e78c061e"}
Nov 02 17:47:29.212865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:29.211674    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" podStartSLOduration=2.211655317 podStartE2EDuration="2.211655317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:29.203484958 +0000 UTC m=+861.782894355" watchObservedRunningTime="2026-11-02 17:47:29.211655317 +0000 UTC m=+861.791064705"
Nov 02 17:47:30.101866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:30.101295    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:30.118193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:30.118163    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:30.154825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:30.154780    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:30.155758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:30.155738    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:47:38.909100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:38.909055    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:47:38.909684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:38.909125    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:40.425525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.425485    5042 generic.go:356] "Generic (PLEG): container finished" podID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerID="e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d" exitCode=0
Nov 02 17:47:40.426046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.425622    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" event={"ID":"634bad02-9a5c-43a4-909b-a6d8b5e8c968","Type":"ContainerDied","Data":"e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"}
Nov 02 17:47:40.426224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.426202    5042 scope.go:122] "RemoveContainer" containerID="e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"
Nov 02 17:47:40.428262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.428229    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:40.428366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.428331    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:40.428444 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.428425    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:40.954868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.953151    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"]
Nov 02 17:47:40.965525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.965493    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:40.973964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.971489    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"]
Nov 02 17:47:40.974383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:40.974358    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:47:41.049828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.049786    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:47:41.049828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.049829    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:47:41.073194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.073164    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:47:41.117863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.114215    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.117863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.114287    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp5lj\" (UniqueName: \"kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.117863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.114384    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.226914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.225899    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.226914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.225963    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp5lj\" (UniqueName: \"kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.226914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.226040    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.227438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.227410    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.228883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.228814    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.237251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.237189    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp5lj\" (UniqueName: \"kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj\") pod \"maas-api-revoke-sub-f1df0b0c-p27ql\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.292136 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.292097    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:41.470717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.470682    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" event={"ID":"634bad02-9a5c-43a4-909b-a6d8b5e8c968","Type":"ContainerStarted","Data":"07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e"}
Nov 02 17:47:41.718903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:41.717553    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"]
Nov 02 17:47:41.756895 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:47:41.747637    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240d54c5_72bb_477e_a175_f5fef9c365f7.slice/crio-c3b35c6c821b6dbb066869260addb18e4d2e1e5690e012a4d41cca83523572b5 WatchSource:0}: Error finding container c3b35c6c821b6dbb066869260addb18e4d2e1e5690e012a4d41cca83523572b5: Status 404 returned error can't find the container with id c3b35c6c821b6dbb066869260addb18e4d2e1e5690e012a4d41cca83523572b5
Nov 02 17:47:42.131916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:42.131870    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:47:42.132186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:42.132159    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:47:42.132326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:42.132224    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 17:47:42.507221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:42.507162    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql" event={"ID":"240d54c5-72bb-477e-a175-f5fef9c365f7","Type":"ContainerStarted","Data":"c4eb3fce5ca8b08f4c5331f59d2a6f9f9cf809bb5fec5e3fb2c7bb157e13f70c"}
Nov 02 17:47:42.507749 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:42.507240    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql" event={"ID":"240d54c5-72bb-477e-a175-f5fef9c365f7","Type":"ContainerStarted","Data":"c3b35c6c821b6dbb066869260addb18e4d2e1e5690e012a4d41cca83523572b5"}
Nov 02 17:47:42.536305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:42.536243    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql" podStartSLOduration=2.53622298 podStartE2EDuration="2.53622298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:42.528749191 +0000 UTC m=+875.108158578" watchObservedRunningTime="2026-11-02 17:47:42.53622298 +0000 UTC m=+875.115632370"
Nov 02 17:47:43.132585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:43.132548    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:47:43.132771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:43.132609    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 17:47:43.517642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:43.517605    5042 generic.go:356] "Generic (PLEG): container finished" podID="240d54c5-72bb-477e-a175-f5fef9c365f7" containerID="c4eb3fce5ca8b08f4c5331f59d2a6f9f9cf809bb5fec5e3fb2c7bb157e13f70c" exitCode=0
Nov 02 17:47:43.518087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:43.517661    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql" event={"ID":"240d54c5-72bb-477e-a175-f5fef9c365f7","Type":"ContainerDied","Data":"c4eb3fce5ca8b08f4c5331f59d2a6f9f9cf809bb5fec5e3fb2c7bb157e13f70c"}
Nov 02 17:47:44.133805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:44.133767    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:47:44.134134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:44.134108    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 17:47:45.168951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.168916    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:47:45.169528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.169498    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 17:47:45.230790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.230738    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:45.359237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.348564    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj\" (UniqueName: \"kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj\") pod \"240d54c5-72bb-477e-a175-f5fef9c365f7\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") "
Nov 02 17:47:45.359237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.354387    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca\") pod \"240d54c5-72bb-477e-a175-f5fef9c365f7\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") "
Nov 02 17:47:45.359237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.354487    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca\") pod \"240d54c5-72bb-477e-a175-f5fef9c365f7\" (UID: \"240d54c5-72bb-477e-a175-f5fef9c365f7\") "
Nov 02 17:47:45.359237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.358273    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca" pod "240d54c5-72bb-477e-a175-f5fef9c365f7" (UID: "240d54c5-72bb-477e-a175-f5fef9c365f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:45.426685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.417482    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca" pod "240d54c5-72bb-477e-a175-f5fef9c365f7" (UID: "240d54c5-72bb-477e-a175-f5fef9c365f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:47:45.426685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.417718    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj" pod "240d54c5-72bb-477e-a175-f5fef9c365f7" (UID: "240d54c5-72bb-477e-a175-f5fef9c365f7"). InnerVolumeSpecName "kube-api-access-dp5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:45.456430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.456392    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:45.456430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.456426    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/240d54c5-72bb-477e-a175-f5fef9c365f7-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:45.456430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.456436    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp5lj\" (UniqueName: \"kubernetes.io/projected/240d54c5-72bb-477e-a175-f5fef9c365f7-kube-api-access-dp5lj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:45.803772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.803727    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql" event={"ID":"240d54c5-72bb-477e-a175-f5fef9c365f7","Type":"ContainerDied","Data":"c3b35c6c821b6dbb066869260addb18e4d2e1e5690e012a4d41cca83523572b5"}
Nov 02 17:47:45.805167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.805144    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3b35c6c821b6dbb066869260addb18e4d2e1e5690e012a4d41cca83523572b5"
Nov 02 17:47:45.805312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:45.804257    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"
Nov 02 17:47:46.141060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:46.140965    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:47:46.141337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:46.141301    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 17:47:47.135902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:47.133200    5042 patch_prober.go:29] interesting pod/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:47:47.135902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:47.133274    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 02 17:47:47.158605 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:47.158560    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:48.602868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:48.574006    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:47:48.602868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:48.574123    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:48.811472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:48.811430    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:48.897057 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:47:48.896954    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:47:49.095322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.095282    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:47:49.121872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.107990    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:47:49.123977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.123952    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:47:49.165086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.164976    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:47:49.496783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.496733    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:47:49.556871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.531663    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:49.556871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.532191    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:49.556871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.533621    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Nov 02 17:47:49.582354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.582310    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:49.929083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:49.928986    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:47:50.168867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:50.167947    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:47:50.168867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:50.168617    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:47:50.544345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:50.542661    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:47:51.664036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:51.663998    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"]
Nov 02 17:47:51.664702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:51.664276    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" containerID="cri-o://593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa" gracePeriod=30
Nov 02 17:47:54.823097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:54.823043    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"]
Nov 02 17:47:54.824057 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:54.824024    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy" containerID="cri-o://07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e" gracePeriod=30
Nov 02 17:47:55.325719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:55.325681    5042 patch_prober.go:29] interesting pod/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:47:55.325930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:55.325735    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:47:57.031882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.031819    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:47:57.034467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.034442    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"]
Nov 02 17:47:57.039129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.039097    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-986864f0-x2grq"]
Nov 02 17:47:57.134737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.134693    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.134969 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.134762    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.134969 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.134817    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135001    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135079    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135149    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135068    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:57.135226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135178    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135077    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:57.135226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135198    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:57.135226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135218    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135308    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert\") pod \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\" (UID: \"dee90570-f1bb-40a4-b8fb-59c3f53c52dc\") "
Nov 02 17:47:57.135535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135452    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:57.135833 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.135809    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:57.136023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.136001    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.136105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.136024    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.136105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.136038    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.136105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.136050    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.136105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.136063    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.138243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.138210    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:57.138243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.138222    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:57.138383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.138225    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "kube-api-access-w2w2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:57.138383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.138308    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo" pod "dee90570-f1bb-40a4-b8fb-59c3f53c52dc" (UID: "dee90570-f1bb-40a4-b8fb-59c3f53c52dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:47:57.237545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.237446    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.237545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.237482    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.237545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.237509    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2w2w\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-kube-api-access-w2w2w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.237545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.237524    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee90570-f1bb-40a4-b8fb-59c3f53c52dc-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:47:57.311918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.311882    5042 generic.go:356] "Generic (PLEG): container finished" podID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerID="593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa" exitCode=0
Nov 02 17:47:57.312100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.311935    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" event={"ID":"dee90570-f1bb-40a4-b8fb-59c3f53c52dc","Type":"ContainerDied","Data":"593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa"}
Nov 02 17:47:57.312100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.312005    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf" event={"ID":"dee90570-f1bb-40a4-b8fb-59c3f53c52dc","Type":"ContainerDied","Data":"e959f931f0619b7718dc7254a374a5eee7b0f2843455e12215d0a8e0948462ef"}
Nov 02 17:47:57.312100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.311939    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"
Nov 02 17:47:57.312100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.312036    5042 scope.go:122] "RemoveContainer" containerID="593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa"
Nov 02 17:47:57.328898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.328815    5042 scope.go:122] "RemoveContainer" containerID="52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"
Nov 02 17:47:57.342601 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.342576    5042 scope.go:122] "RemoveContainer" containerID="593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa"
Nov 02 17:47:57.343006 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:47:57.342970    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa\": container with ID starting with 593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa not found: ID does not exist" containerID="593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa"
Nov 02 17:47:57.343123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.343003    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa"} err="failed to get container status \"593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa\": rpc error: code = NotFound desc = could not find container \"593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa\": container with ID starting with 593e99aee220987d34d4f733dd8a03dd0d6cd0611adc2a7f5d7bc0f50e5d0daa not found: ID does not exist"
Nov 02 17:47:57.343123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.343024    5042 scope.go:122] "RemoveContainer" containerID="52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"
Nov 02 17:47:57.343352 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:47:57.343326    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259\": container with ID starting with 52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259 not found: ID does not exist" containerID="52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"
Nov 02 17:47:57.343474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.343362    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259"} err="failed to get container status \"52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259\": rpc error: code = NotFound desc = could not find container \"52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259\": container with ID starting with 52b6314a19284b32087e499386784400308f758cf7eb01e429fa2bfc37009259 not found: ID does not exist"
Nov 02 17:47:57.345277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.345255    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"]
Nov 02 17:47:57.350694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.350672    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2893d1-openshift-default-546f79d79-hxgpf"]
Nov 02 17:47:57.549790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.549707    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352b62f2-aa75-44d4-a7a5-62f3067a5bb8" path="/var/lib/kubelet/pods/352b62f2-aa75-44d4-a7a5-62f3067a5bb8/volumes"
Nov 02 17:47:57.550716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.550689    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" path="/var/lib/kubelet/pods/dee90570-f1bb-40a4-b8fb-59c3f53c52dc/volumes"
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.768256    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"]
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.769227    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy"
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.769246    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy"
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.769303    5042 cpu_manager.go:414] "containerMap: removing container" podUID="240d54c5-72bb-477e-a175-f5fef9c365f7" containerName="revoke-keys"
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.769311    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="240d54c5-72bb-477e-a175-f5fef9c365f7" containerName="revoke-keys"
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.769479    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy"
Nov 02 17:47:57.772992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.769504    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="240d54c5-72bb-477e-a175-f5fef9c365f7" containerName="revoke-keys"
Nov 02 17:47:57.775584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.775556    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.783077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.783054    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-11f0a3-openshift-default-dockercfg-pthvj\""
Nov 02 17:47:57.792800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.792763    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"]
Nov 02 17:47:57.847096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.846891    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.847578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.847539    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.847707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.847682    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.847758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.847741    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.849331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.847825    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.849331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.847979    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.849331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.848067    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.849331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.848137    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.849331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.848192    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxvp\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949328 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949289    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949365    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949416    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949447    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxvp\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949516    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949534    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949599    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949640    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949712    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.949966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949744    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.950024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.949972    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.950081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.950036    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.950207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.950186    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.950522 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.950498    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.952100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.952078    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.952213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.952191    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.959017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.958997    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:57.959235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:57.959217    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxvp\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp\") pod \"e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") " pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:58.117909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:58.117140    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:47:58.543924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:58.526797    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"]
Nov 02 17:47:58.547441 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:47:58.547411    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ae370c_b74a_4501_828e_414e0e8471d1.slice/crio-9432d913081313e280b9e2d5433e72432ace9740412f95e34421d5e300240a0d WatchSource:0}: Error finding container 9432d913081313e280b9e2d5433e72432ace9740412f95e34421d5e300240a0d: Status 404 returned error can't find the container with id 9432d913081313e280b9e2d5433e72432ace9740412f95e34421d5e300240a0d
Nov 02 17:47:58.562493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:58.561213    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:58.562493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:58.561320    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:58.562493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:58.561372    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:59.339648 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:59.339612    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" event={"ID":"e0ae370c-b74a-4501-828e-414e0e8471d1","Type":"ContainerStarted","Data":"0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"}
Nov 02 17:47:59.340130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:59.339703    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" event={"ID":"e0ae370c-b74a-4501-828e-414e0e8471d1","Type":"ContainerStarted","Data":"9432d913081313e280b9e2d5433e72432ace9740412f95e34421d5e300240a0d"}
Nov 02 17:47:59.367591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:47:59.367495    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podStartSLOduration=2.367480721 podStartE2EDuration="2.367480721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:59.363573927 +0000 UTC m=+891.942983313" watchObservedRunningTime="2026-11-02 17:47:59.367480721 +0000 UTC m=+891.946890107"
Nov 02 17:48:00.118239 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.118160    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:00.134783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.134754    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:00.181750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.181722    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:48:00.207832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.207799    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.208425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.208402    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.208880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.208863    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.209296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.209278    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.209800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.208340    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:00.209901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.208760    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:00.209901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.209182    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:00.209901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.209682    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:00.210060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.210044    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.210132 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.210094    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.210191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.210142    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.210524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.210274    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.210524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.210366    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo\") pod \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\" (UID: \"634bad02-9a5c-43a4-909b-a6d8b5e8c968\") "
Nov 02 17:48:00.211189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.211167    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:00.211488 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.211466    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.211582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.211493    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.211582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.211506    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.211582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.211519    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.211582 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.211533    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.213214 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.213188    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "kube-api-access-k7zmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:00.214243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.214214    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:00.215228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.215059    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:00.215597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.215574    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo" pod "634bad02-9a5c-43a4-909b-a6d8b5e8c968" (UID: "634bad02-9a5c-43a4-909b-a6d8b5e8c968"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:48:00.346880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.320589    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.346880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.320624    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.346880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.320638    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7zmx\" (UniqueName: \"kubernetes.io/projected/634bad02-9a5c-43a4-909b-a6d8b5e8c968-kube-api-access-k7zmx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.346880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.320652    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/634bad02-9a5c-43a4-909b-a6d8b5e8c968-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:00.444996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.436305    5042 generic.go:356] "Generic (PLEG): container finished" podID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerID="07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e" exitCode=0
Nov 02 17:48:00.444996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.441736    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"
Nov 02 17:48:00.455208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.448091    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" event={"ID":"634bad02-9a5c-43a4-909b-a6d8b5e8c968","Type":"ContainerDied","Data":"07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e"}
Nov 02 17:48:00.455208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.448167    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp" event={"ID":"634bad02-9a5c-43a4-909b-a6d8b5e8c968","Type":"ContainerDied","Data":"e28c621f8620ec3c57d5dd152cfdfd0ed4085677abf2b3dfd566aa0786cbffd3"}
Nov 02 17:48:00.455208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.448203    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:00.455208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.448234    5042 scope.go:122] "RemoveContainer" containerID="07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e"
Nov 02 17:48:00.476478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.475294    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:00.485327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.485299    5042 scope.go:122] "RemoveContainer" containerID="e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"
Nov 02 17:48:00.512545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.512527    5042 scope.go:122] "RemoveContainer" containerID="07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e"
Nov 02 17:48:00.512918 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:00.512898    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e\": container with ID starting with 07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e not found: ID does not exist" containerID="07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e"
Nov 02 17:48:00.513019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.512925    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e"} err="failed to get container status \"07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e\": rpc error: code = NotFound desc = could not find container \"07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e\": container with ID starting with 07a2087fb70f7e744c998ef1b6af7575da227946421f4c2344537e70aa00bf7e not found: ID does not exist"
Nov 02 17:48:00.513019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.512961    5042 scope.go:122] "RemoveContainer" containerID="e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"
Nov 02 17:48:00.513352 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:00.513282    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d\": container with ID starting with e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d not found: ID does not exist" containerID="e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"
Nov 02 17:48:00.513352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.513311    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d"} err="failed to get container status \"e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d\": rpc error: code = NotFound desc = could not find container \"e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d\": container with ID starting with e9b9390d482d730476f5bf166dfecab7d7c300ffacf4e1c70a7dee28833d559d not found: ID does not exist"
Nov 02 17:48:00.537079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.537046    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"]
Nov 02 17:48:00.566685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:00.566651    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-58970f-openshift-default-bd89b5d7f-gk4zp"]
Nov 02 17:48:01.545120 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:01.545083    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" path="/var/lib/kubelet/pods/634bad02-9a5c-43a4-909b-a6d8b5e8c968/volumes"
Nov 02 17:48:05.945334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.945300    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"]
Nov 02 17:48:05.946158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946137    5042 cpu_manager.go:414] "containerMap: removing container" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy"
Nov 02 17:48:05.946158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946158    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy"
Nov 02 17:48:05.946321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946190    5042 cpu_manager.go:414] "containerMap: removing container" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy"
Nov 02 17:48:05.946321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946198    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy"
Nov 02 17:48:05.946321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946255    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy"
Nov 02 17:48:05.946321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946263    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy"
Nov 02 17:48:05.946584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946429    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy"
Nov 02 17:48:05.946584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946447    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee90570-f1bb-40a4-b8fb-59c3f53c52dc" containerName="istio-proxy"
Nov 02 17:48:05.946584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.946461    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="634bad02-9a5c-43a4-909b-a6d8b5e8c968" containerName="istio-proxy"
Nov 02 17:48:05.952734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.952702    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:05.959105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.959072    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-da5f07e6-openshift-default-dockercfg-hpwn4\""
Nov 02 17:48:05.994955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:05.994919    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"]
Nov 02 17:48:06.047050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.047008    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.053267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.053229    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.053545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.053527    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.053747 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.053732    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.054016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.053997    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.054147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.054132    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.054277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.054263    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4nf\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.054408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.054395    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.054497 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.054485    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.155879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.155812    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.156674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.156656    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.156949    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4nf\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157024    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157032    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157057    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157179    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157268    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157295    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.156611    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.157649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157414    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.158501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157819    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.158501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.157874    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.158501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.158053    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.159645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.159607    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.160208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.160193    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.166654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.166630    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.167212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.167178    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4nf\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf\") pod \"e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") " pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.312458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.311112    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:06.718631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.718594    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:06.719731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.719703    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:06.720397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.720200    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:06.720397 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:06.720326    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"]
Nov 02 17:48:07.671267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:07.671231    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" event={"ID":"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1","Type":"ContainerStarted","Data":"9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae"}
Nov 02 17:48:07.671930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:07.671812    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" event={"ID":"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1","Type":"ContainerStarted","Data":"a4dadc692f76742faca8e7645d15286f76f7b21da3a74d68e2f64f72fb0b2127"}
Nov 02 17:48:07.708072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:07.708016    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podStartSLOduration=2.707999155 podStartE2EDuration="2.707999155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:07.695403224 +0000 UTC m=+900.274812611" watchObservedRunningTime="2026-11-02 17:48:07.707999155 +0000 UTC m=+900.287408622"
Nov 02 17:48:08.035242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.035199    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"]
Nov 02 17:48:08.035576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.035551    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:48:08.043831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.043779    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"]
Nov 02 17:48:08.044385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.044365    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:48:08.052697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.052286    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ead5a273-p28bf"]
Nov 02 17:48:08.119891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.115906    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715178e5-jqq78"]
Nov 02 17:48:08.186883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.186834    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:48:08.195983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.190810    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:48:08.315249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.315091    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:08.318905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.317041    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:08.318905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.317112    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:08.417862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.417683    5042 scope.go:122] "RemoveContainer" containerID="5443cf15779a25447c460d79917b782b26e3485baab1951efeea161b57a6e075"
Nov 02 17:48:08.452130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.451732    5042 scope.go:122] "RemoveContainer" containerID="9ff97b7e40a65a28103e87081232c726e1c5866b290e58b5b6043cb59ccd3efa"
Nov 02 17:48:08.500405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.498030    5042 scope.go:122] "RemoveContainer" containerID="892f80d7e36948497b7f6806eede06671cd69fabeae04f21c8e5755f685743ac"
Nov 02 17:48:08.728802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:08.728770    5042 scope.go:122] "RemoveContainer" containerID="bd2695947087fe83af782996d333fda0f06b5b4a6253fba1db30c21de65fd772"
Nov 02 17:48:09.314893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:09.312232    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:09.314893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:09.312334    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:09.556269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:09.555395    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636f30cc-655b-4e47-9d3b-c48c980db943" path="/var/lib/kubelet/pods/636f30cc-655b-4e47-9d3b-c48c980db943/volumes"
Nov 02 17:48:09.556269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:09.555936    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7df942-dcc0-4a93-ba77-2b1b8e86995a" path="/var/lib/kubelet/pods/ab7df942-dcc0-4a93-ba77-2b1b8e86995a/volumes"
Nov 02 17:48:10.323870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:10.320477    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:10.323870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:10.320571    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:10.451282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:10.447712    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"]
Nov 02 17:48:10.451282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:10.448132    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" podUID="55a52f6d-c279-4d1a-b201-66d2e1e75109" containerName="istio-proxy" containerID="cri-o://d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf" gracePeriod=30
Nov 02 17:48:11.313864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:11.313502    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:11.313864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:11.313562    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:12.050358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:12.049908    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"]
Nov 02 17:48:12.056876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:12.056826    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10430d88-hf7lb"]
Nov 02 17:48:12.320923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:12.315282    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:12.320923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:12.315342    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:13.313481 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:13.313444    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:13.314073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:13.314039    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:13.591467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:13.591039    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="becbac2b-a694-424d-b523-17d339d879f4" path="/var/lib/kubelet/pods/becbac2b-a694-424d-b523-17d339d879f4/volumes"
Nov 02 17:48:14.319865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.314226    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Nov 02 17:48:14.319865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.314289    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Nov 02 17:48:14.460866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.459995    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:14.539858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.539793    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:14.554194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.554139    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:14.952533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.952481    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:48:14.997066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:14.997003    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:15.466874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:15.462183    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:15.466874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:15.462830    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:48:15.605902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:15.594708    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:15.614884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:15.606510    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:15.651511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:15.629467    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:15.999251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:15.999204    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:48:16.036367 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.036311    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.142961    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143077    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143140    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143187    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143232    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143283    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143326    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143378    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:48:16.146291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.143419    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:48:16.154546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.149080    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:16.154546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.152567    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:48:16.179864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.177793    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"]
Nov 02 17:48:16.241864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.240530    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdc3543f-dffj4"]
Nov 02 17:48:16.289078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:16.267663    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:48:17.548320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:17.548276    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac" path="/var/lib/kubelet/pods/0bb86630-8d96-4cbe-9eb4-2b6f9fee96ac/volumes"
Nov 02 17:48:18.542307 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:18.542266    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:48:18.646172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:18.644517    5042 patch_prober.go:29] interesting pod/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": read tcp 10.132.0.2:48854->10.132.0.120:15021: read: connection reset by peer" start-of-body=
Nov 02 17:48:18.646172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:18.644598    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" podUID="55a52f6d-c279-4d1a-b201-66d2e1e75109" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": read tcp 10.132.0.2:48854->10.132.0.120:15021: read: connection reset by peer"
Nov 02 17:48:18.942513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:18.942486    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:48:19.041816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.041774    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042064 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.041822    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042064 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.041924    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042064 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.041965    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042064 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042015    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042113    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042173    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042215    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042224    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:19.042473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042317    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo\") pod \"55a52f6d-c279-4d1a-b201-66d2e1e75109\" (UID: \"55a52f6d-c279-4d1a-b201-66d2e1e75109\") "
Nov 02 17:48:19.042473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042331    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:19.042585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042488    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:19.042585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042504    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:19.043079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.042993    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:19.043079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.043071    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.043079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.043091    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.043079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.043106    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.043079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.043117    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.044340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.044320    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "kube-api-access-wztmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:19.044722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.044707    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:48:19.044952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.044922    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:19.045044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.044964    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy" pod "55a52f6d-c279-4d1a-b201-66d2e1e75109" (UID: "55a52f6d-c279-4d1a-b201-66d2e1e75109"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:19.144215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.144125    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wztmc\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-kube-api-access-wztmc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.144215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.144160    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.144215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.144172    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.144215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.144181    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a52f6d-c279-4d1a-b201-66d2e1e75109-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.144215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.144189    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a52f6d-c279-4d1a-b201-66d2e1e75109-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:19.278481 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.278439    5042 generic.go:356] "Generic (PLEG): container finished" podID="55a52f6d-c279-4d1a-b201-66d2e1e75109" containerID="d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf" exitCode=0
Nov 02 17:48:19.278681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.278559    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"
Nov 02 17:48:19.278681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.278591    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" event={"ID":"55a52f6d-c279-4d1a-b201-66d2e1e75109","Type":"ContainerDied","Data":"d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf"}
Nov 02 17:48:19.278681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.278673    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8" event={"ID":"55a52f6d-c279-4d1a-b201-66d2e1e75109","Type":"ContainerDied","Data":"452485934c8d7091a6ebcf8d63590d14d575128a74a939901220cb637c44b757"}
Nov 02 17:48:19.278879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.278696    5042 scope.go:122] "RemoveContainer" containerID="d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf"
Nov 02 17:48:19.291326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.291303    5042 scope.go:122] "RemoveContainer" containerID="d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf"
Nov 02 17:48:19.291657 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:19.291638    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf\": container with ID starting with d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf not found: ID does not exist" containerID="d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf"
Nov 02 17:48:19.291726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.291668    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf"} err="failed to get container status \"d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf\": rpc error: code = NotFound desc = could not find container \"d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf\": container with ID starting with d1b0d642905569f2f65423afee71731642319324b68c6116a8f2fb2d635469bf not found: ID does not exist"
Nov 02 17:48:19.306419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.306391    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"]
Nov 02 17:48:19.311084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.311057    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2129b9-openshift-default-6d96dc7649-2gxc8"]
Nov 02 17:48:19.546891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:19.546859    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a52f6d-c279-4d1a-b201-66d2e1e75109" path="/var/lib/kubelet/pods/55a52f6d-c279-4d1a-b201-66d2e1e75109/volumes"
Nov 02 17:48:20.038786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:20.038752    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"]
Nov 02 17:48:20.044330 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:20.044301    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9c47287-rvjnj"]
Nov 02 17:48:21.039460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:21.039398    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"]
Nov 02 17:48:21.044153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:21.044125    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a41480aa-j4d8b"]
Nov 02 17:48:21.548653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:21.548616    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012ec9b3-5eed-4bbd-a0fb-54144a80e8d8" path="/var/lib/kubelet/pods/012ec9b3-5eed-4bbd-a0fb-54144a80e8d8/volumes"
Nov 02 17:48:21.549398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:21.549371    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd775878-72e3-43d4-8a09-562005933027" path="/var/lib/kubelet/pods/fd775878-72e3-43d4-8a09-562005933027/volumes"
Nov 02 17:48:22.502715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.502680    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"]
Nov 02 17:48:22.503324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.503309    5042 cpu_manager.go:414] "containerMap: removing container" podUID="55a52f6d-c279-4d1a-b201-66d2e1e75109" containerName="istio-proxy"
Nov 02 17:48:22.503385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.503325    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a52f6d-c279-4d1a-b201-66d2e1e75109" containerName="istio-proxy"
Nov 02 17:48:22.503474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.503464    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a52f6d-c279-4d1a-b201-66d2e1e75109" containerName="istio-proxy"
Nov 02 17:48:22.507510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.507485    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.515453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.515421    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:48:22.515685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.515662    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"]
Nov 02 17:48:22.517088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.516881    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:48:22.517088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.516904    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:48:22.517088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.516962    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:48:22.603478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.603435    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmm9\" (UniqueName: \"kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.603672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.603508    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.603752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.603725    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.705117 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.705078    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmm9\" (UniqueName: \"kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.705328 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.705151    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.705819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.705374    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.706143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.706123    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.708023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.708003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.714213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.714192    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmm9\" (UniqueName: \"kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9\") pod \"maas-api-revoke-sub-690925a8-xjdzj\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:22.853571 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:22.853472    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:23.151725 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:48:23.151687    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976d7ec1_033a_48b7_b75a_8459e7a94e5c.slice/crio-17c3a6461a1bbbed53f65e2ceffbfb76b6f5cad92fdad3a83ad50dc9d189c59b WatchSource:0}: Error finding container 17c3a6461a1bbbed53f65e2ceffbfb76b6f5cad92fdad3a83ad50dc9d189c59b: Status 404 returned error can't find the container with id 17c3a6461a1bbbed53f65e2ceffbfb76b6f5cad92fdad3a83ad50dc9d189c59b
Nov 02 17:48:23.152137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:23.152113    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"]
Nov 02 17:48:23.374333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:23.374250    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj" event={"ID":"976d7ec1-033a-48b7-b75a-8459e7a94e5c","Type":"ContainerStarted","Data":"17c3a6461a1bbbed53f65e2ceffbfb76b6f5cad92fdad3a83ad50dc9d189c59b"}
Nov 02 17:48:24.384140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:24.384050    5042 generic.go:356] "Generic (PLEG): container finished" podID="976d7ec1-033a-48b7-b75a-8459e7a94e5c" containerID="7924bfe2f3ef74a92f35b551ed10b8766f1460182b8912daa1acaf7ddecf870e" exitCode=0
Nov 02 17:48:24.384140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:24.384109    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj" event={"ID":"976d7ec1-033a-48b7-b75a-8459e7a94e5c","Type":"ContainerDied","Data":"7924bfe2f3ef74a92f35b551ed10b8766f1460182b8912daa1acaf7ddecf870e"}
Nov 02 17:48:25.607548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.607520    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:25.762115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.762076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca\") pod \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") "
Nov 02 17:48:25.762329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.762149    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca\") pod \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") "
Nov 02 17:48:25.762329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.762239    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9\" (UniqueName: \"kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9\") pod \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\" (UID: \"976d7ec1-033a-48b7-b75a-8459e7a94e5c\") "
Nov 02 17:48:25.762505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.762477    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca" pod "976d7ec1-033a-48b7-b75a-8459e7a94e5c" (UID: "976d7ec1-033a-48b7-b75a-8459e7a94e5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:25.763016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.762989    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:25.764882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.764822    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9" pod "976d7ec1-033a-48b7-b75a-8459e7a94e5c" (UID: "976d7ec1-033a-48b7-b75a-8459e7a94e5c"). InnerVolumeSpecName "kube-api-access-ldmm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:25.764983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.764966    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca" pod "976d7ec1-033a-48b7-b75a-8459e7a94e5c" (UID: "976d7ec1-033a-48b7-b75a-8459e7a94e5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:48:25.863915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.863881    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/976d7ec1-033a-48b7-b75a-8459e7a94e5c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:25.863915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:25.863906    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldmm9\" (UniqueName: \"kubernetes.io/projected/976d7ec1-033a-48b7-b75a-8459e7a94e5c-kube-api-access-ldmm9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:26.402272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:26.402241    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"
Nov 02 17:48:26.402451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:26.402270    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj" event={"ID":"976d7ec1-033a-48b7-b75a-8459e7a94e5c","Type":"ContainerDied","Data":"17c3a6461a1bbbed53f65e2ceffbfb76b6f5cad92fdad3a83ad50dc9d189c59b"}
Nov 02 17:48:26.402451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:26.402325    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c3a6461a1bbbed53f65e2ceffbfb76b6f5cad92fdad3a83ad50dc9d189c59b"
Nov 02 17:48:27.306185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.306141    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"]
Nov 02 17:48:27.307134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.307115    5042 cpu_manager.go:414] "containerMap: removing container" podUID="976d7ec1-033a-48b7-b75a-8459e7a94e5c" containerName="revoke-keys"
Nov 02 17:48:27.307186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.307137    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976d7ec1-033a-48b7-b75a-8459e7a94e5c" containerName="revoke-keys"
Nov 02 17:48:27.307372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.307360    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="976d7ec1-033a-48b7-b75a-8459e7a94e5c" containerName="revoke-keys"
Nov 02 17:48:27.312041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.312019    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.320111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.320084    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:48:27.322516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.322488    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:48:27.326107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.326082    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"]
Nov 02 17:48:27.328591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.328567    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:48:27.328591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.328579    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:48:27.390504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.390473    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.390652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.390625    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79f9\" (UniqueName: \"kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.390714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.390650    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.492205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.492166    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.492366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.492295    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79f9\" (UniqueName: \"kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.492366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.492316    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.493116 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.493053    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.495074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.495045    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.500905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.500879    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79f9\" (UniqueName: \"kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9\") pod \"maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.629735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.629655    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:27.923922 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:48:27.923834    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500aef43_d22f_40d2_a5ba_9f840d722a98.slice/crio-ad0d8ba05f0110252a818ca81f753e9f08bfe8300c75043ca514ac1f454af1c8 WatchSource:0}: Error finding container ad0d8ba05f0110252a818ca81f753e9f08bfe8300c75043ca514ac1f454af1c8: Status 404 returned error can't find the container with id ad0d8ba05f0110252a818ca81f753e9f08bfe8300c75043ca514ac1f454af1c8
Nov 02 17:48:27.926270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:27.926233    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"]
Nov 02 17:48:28.426815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:28.426764    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk" event={"ID":"500aef43-d22f-40d2-a5ba-9f840d722a98","Type":"ContainerStarted","Data":"3223110529f067a6e7edeac575521329fb9b21957735a3dfbe7508945baeec5b"}
Nov 02 17:48:28.427247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:28.426828    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk" event={"ID":"500aef43-d22f-40d2-a5ba-9f840d722a98","Type":"ContainerStarted","Data":"ad0d8ba05f0110252a818ca81f753e9f08bfe8300c75043ca514ac1f454af1c8"}
Nov 02 17:48:28.446861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:28.446788    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk" podStartSLOduration=1.446773869 podStartE2EDuration="1.446773869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:28.445083502 +0000 UTC m=+921.024492891" watchObservedRunningTime="2026-11-02 17:48:28.446773869 +0000 UTC m=+921.026183254"
Nov 02 17:48:29.437259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:29.437211    5042 generic.go:356] "Generic (PLEG): container finished" podID="500aef43-d22f-40d2-a5ba-9f840d722a98" containerID="3223110529f067a6e7edeac575521329fb9b21957735a3dfbe7508945baeec5b" exitCode=0
Nov 02 17:48:29.437701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:29.437293    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk" event={"ID":"500aef43-d22f-40d2-a5ba-9f840d722a98","Type":"ContainerDied","Data":"3223110529f067a6e7edeac575521329fb9b21957735a3dfbe7508945baeec5b"}
Nov 02 17:48:30.721534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.721163    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:30.821866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.821511    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9\" (UniqueName: \"kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9\") pod \"500aef43-d22f-40d2-a5ba-9f840d722a98\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") "
Nov 02 17:48:30.821866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.821682    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca\") pod \"500aef43-d22f-40d2-a5ba-9f840d722a98\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") "
Nov 02 17:48:30.821866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.821770    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca\") pod \"500aef43-d22f-40d2-a5ba-9f840d722a98\" (UID: \"500aef43-d22f-40d2-a5ba-9f840d722a98\") "
Nov 02 17:48:30.825800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.825004    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9" pod "500aef43-d22f-40d2-a5ba-9f840d722a98" (UID: "500aef43-d22f-40d2-a5ba-9f840d722a98"). InnerVolumeSpecName "kube-api-access-c79f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:30.826620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.826118    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca" pod "500aef43-d22f-40d2-a5ba-9f840d722a98" (UID: "500aef43-d22f-40d2-a5ba-9f840d722a98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:30.831345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.831297    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca" pod "500aef43-d22f-40d2-a5ba-9f840d722a98" (UID: "500aef43-d22f-40d2-a5ba-9f840d722a98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:48:30.924667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.924624    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c79f9\" (UniqueName: \"kubernetes.io/projected/500aef43-d22f-40d2-a5ba-9f840d722a98-kube-api-access-c79f9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:30.924667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.924666    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:30.924920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:30.924684    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500aef43-d22f-40d2-a5ba-9f840d722a98-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:31.472005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:31.471973    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"
Nov 02 17:48:31.485864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:31.472118    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk" event={"ID":"500aef43-d22f-40d2-a5ba-9f840d722a98","Type":"ContainerDied","Data":"ad0d8ba05f0110252a818ca81f753e9f08bfe8300c75043ca514ac1f454af1c8"}
Nov 02 17:48:31.485864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:31.485029    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad0d8ba05f0110252a818ca81f753e9f08bfe8300c75043ca514ac1f454af1c8"
Nov 02 17:48:32.034772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:32.034736    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"]
Nov 02 17:48:32.039191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:32.039161    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e9ecf52-lj5nq"]
Nov 02 17:48:33.566542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:33.566504    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764ed14c-791a-4f72-bffc-3a60e6f65235" path="/var/lib/kubelet/pods/764ed14c-791a-4f72-bffc-3a60e6f65235/volumes"
Nov 02 17:48:38.014490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:38.014419    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:38.273491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:38.273375    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:38.553933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:38.535963    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:38.553933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:38.536518    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:48:39.033864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.030474    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:39.065898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.055916    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Nov 02 17:48:39.085929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.085300    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:39.364208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.362931    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:39.364208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.363232    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:48:39.493251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.492947    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:39.559874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.536480    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:39.559874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.536634    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:39.559874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.537003    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:48:39.559874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.537063    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:48:39.559874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:39.550877    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:40.029873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.018954    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:48:40.089962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.089191    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:40.093136 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.091460    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:48:40.388921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.372996    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:40.439868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.432384    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:48:40.546876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.527390    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:48:40.551289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.547159    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:40.552126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.551536    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:40.576323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.552596    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:48:40.600885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.591637    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:40.603065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:40.603037    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:48:41.029173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:41.029131    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:48:41.030408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:41.030383    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:48:41.104868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:41.087464    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:48:41.168039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:41.163272    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:48:41.243905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:41.226248    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:48:43.155876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.086361    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"]
Nov 02 17:48:43.155876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.087380    5042 cpu_manager.go:414] "containerMap: removing container" podUID="500aef43-d22f-40d2-a5ba-9f840d722a98" containerName="revoke-keys"
Nov 02 17:48:43.155876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.087400    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500aef43-d22f-40d2-a5ba-9f840d722a98" containerName="revoke-keys"
Nov 02 17:48:43.155876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.087616    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="500aef43-d22f-40d2-a5ba-9f840d722a98" containerName="revoke-keys"
Nov 02 17:48:43.155876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.131407    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.163904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.160916    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"]
Nov 02 17:48:43.167075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.167051    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.171305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.169475    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"]
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.181058    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.181311    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.181349    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182159    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182321    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182359    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182420    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182519    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182655    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182698    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb85w\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182785    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182812    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.199865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.182887    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.231920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.228927    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"]
Nov 02 17:48:43.302052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.296795    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.302052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.301998    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302131    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.302316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302189    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302230    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb85w\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302325    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302353    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302412    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302492    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302520    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.302970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302547    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtl6m\" (UniqueName: \"kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.302970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.302614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.303275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.303239    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.304163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.304137    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.304293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.304159    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.304394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.304230    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.304672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.304632    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.306610 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.306585    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.306708 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.306688    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.318993 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.318967    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb85w\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.357513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.335017    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token\") pod \"e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") " pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.406799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.406704    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtl6m\" (UniqueName: \"kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.414866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.407143    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.415294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.415259    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.428869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.417222    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.433249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.433218    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.439876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.434910    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtl6m\" (UniqueName: \"kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m\") pod \"maas-api-revoke-sub-c1683322-94jtq\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.458624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.458591    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:43.543220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.542196    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:43.828071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.828017    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"]
Nov 02 17:48:43.842719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.842680    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:43.842859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.842792    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:43.842916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.842891    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:43.955123 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:48:43.955090    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dd0e5d_081f_4826_a542_c7388276bf55.slice/crio-0fdb2e11399145facb70e74d6d62f599f3623353b14043d9d116b1fee3cd5e43 WatchSource:0}: Error finding container 0fdb2e11399145facb70e74d6d62f599f3623353b14043d9d116b1fee3cd5e43: Status 404 returned error can't find the container with id 0fdb2e11399145facb70e74d6d62f599f3623353b14043d9d116b1fee3cd5e43
Nov 02 17:48:43.957457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:43.957432    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"]
Nov 02 17:48:44.094324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.094270    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq" event={"ID":"f6dd0e5d-081f-4826-a542-c7388276bf55","Type":"ContainerStarted","Data":"0fdb2e11399145facb70e74d6d62f599f3623353b14043d9d116b1fee3cd5e43"}
Nov 02 17:48:44.108178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.108125    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" event={"ID":"a251a508-27db-49b1-8832-75681664850e","Type":"ContainerStarted","Data":"f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801"}
Nov 02 17:48:44.108351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.108196    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" event={"ID":"a251a508-27db-49b1-8832-75681664850e","Type":"ContainerStarted","Data":"dade2c6297afd5a24765c3e580702f9a81f56a33e9d4378ddd9cb776f2a55a19"}
Nov 02 17:48:44.145950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.145853    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" podStartSLOduration=2.145818332 podStartE2EDuration="2.145818332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:44.142235791 +0000 UTC m=+936.721645188" watchObservedRunningTime="2026-11-02 17:48:44.145818332 +0000 UTC m=+936.725227721"
Nov 02 17:48:44.459902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.459236    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:44.461234 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.461195    5042 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 17:48:44.461460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:44.461264    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 17:48:45.127223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:45.127183    5042 generic.go:356] "Generic (PLEG): container finished" podID="f6dd0e5d-081f-4826-a542-c7388276bf55" containerID="72976f457df52ea8681039ab008de3d47cc5b24cf011b17f8db8453218206d77" exitCode=0
Nov 02 17:48:45.127382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:45.127324    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq" event={"ID":"f6dd0e5d-081f-4826-a542-c7388276bf55","Type":"ContainerDied","Data":"72976f457df52ea8681039ab008de3d47cc5b24cf011b17f8db8453218206d77"}
Nov 02 17:48:45.459631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:45.459592    5042 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 17:48:45.459807 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:45.459674    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 17:48:46.197315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.196956    5042 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:48:46.197315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.197033    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" podUID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:46.459950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.459914    5042 patch_prober.go:29] interesting pod/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Nov 02 17:48:46.460142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.459973    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Nov 02 17:48:46.487915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.480140    5042 patch_prober.go:29] interesting pod/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:48:46.487915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.480205    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:46.566452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.566410    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:46.626061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.626018    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"]
Nov 02 17:48:46.627179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.627146    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" podUID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" containerName="istio-proxy" containerID="cri-o://3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d" gracePeriod=30
Nov 02 17:48:46.644919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.644886    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"]
Nov 02 17:48:46.743206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.743029    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca\") pod \"f6dd0e5d-081f-4826-a542-c7388276bf55\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") "
Nov 02 17:48:46.743392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.743315    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca\") pod \"f6dd0e5d-081f-4826-a542-c7388276bf55\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") "
Nov 02 17:48:46.743392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.743382    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m\" (UniqueName: \"kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m\") pod \"f6dd0e5d-081f-4826-a542-c7388276bf55\" (UID: \"f6dd0e5d-081f-4826-a542-c7388276bf55\") "
Nov 02 17:48:46.744754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.744724    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca" pod "f6dd0e5d-081f-4826-a542-c7388276bf55" (UID: "f6dd0e5d-081f-4826-a542-c7388276bf55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:46.748908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.748875    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m" pod "f6dd0e5d-081f-4826-a542-c7388276bf55" (UID: "f6dd0e5d-081f-4826-a542-c7388276bf55"). InnerVolumeSpecName "kube-api-access-gtl6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:46.749741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.749714    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca" pod "f6dd0e5d-081f-4826-a542-c7388276bf55" (UID: "f6dd0e5d-081f-4826-a542-c7388276bf55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:48:46.860941 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.846853    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:46.860941 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.846892    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6dd0e5d-081f-4826-a542-c7388276bf55-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:46.860941 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:46.846908    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtl6m\" (UniqueName: \"kubernetes.io/projected/f6dd0e5d-081f-4826-a542-c7388276bf55-kube-api-access-gtl6m\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:47.238862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.238354    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy" containerID="cri-o://f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801" gracePeriod=30
Nov 02 17:48:47.238862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.238678    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"
Nov 02 17:48:47.246295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.246252    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq" event={"ID":"f6dd0e5d-081f-4826-a542-c7388276bf55","Type":"ContainerDied","Data":"0fdb2e11399145facb70e74d6d62f599f3623353b14043d9d116b1fee3cd5e43"}
Nov 02 17:48:47.246618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.246593    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fdb2e11399145facb70e74d6d62f599f3623353b14043d9d116b1fee3cd5e43"
Nov 02 17:48:47.750587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.750556    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"]
Nov 02 17:48:47.752699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.751413    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f6dd0e5d-081f-4826-a542-c7388276bf55" containerName="revoke-keys"
Nov 02 17:48:47.752699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.751431    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6dd0e5d-081f-4826-a542-c7388276bf55" containerName="revoke-keys"
Nov 02 17:48:47.752699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.751631    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6dd0e5d-081f-4826-a542-c7388276bf55" containerName="revoke-keys"
Nov 02 17:48:47.802864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.796047    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"]
Nov 02 17:48:47.802864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.796214    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:47.807861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.803493    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4c9298\""
Nov 02 17:48:47.807861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.804249    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4c9298-dockercfg-2qhkz\""
Nov 02 17:48:47.892858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.891893    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:47.909155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:47.907211    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m74mn\" (UniqueName: \"kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn\") pod \"payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:48.011400 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:48.009940    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:48.011400 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:48.010162    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m74mn\" (UniqueName: \"kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn\") pod \"payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:48.017192 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:48.017155    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:48.025162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:48.025123    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m74mn\" (UniqueName: \"kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn\") pod \"payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:48.146290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:48.146245    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:48.845988 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:48.841735    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:48:48.940325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:48.939690    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"]
Nov 02 17:48:48.944196 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:48:48.944012    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181223b0_54c0_4917_ae3c_db275156931c.slice/crio-ad1eee10120dbf309180b0326d6007560408b2584ab7624ff71e1ff9805371e6 WatchSource:0}: Error finding container ad1eee10120dbf309180b0326d6007560408b2584ab7624ff71e1ff9805371e6: Status 404 returned error can't find the container with id ad1eee10120dbf309180b0326d6007560408b2584ab7624ff71e1ff9805371e6
Nov 02 17:48:49.299272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:49.299235    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44" event={"ID":"181223b0-54c0-4917-ae3c-db275156931c","Type":"ContainerStarted","Data":"ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b"}
Nov 02 17:48:49.299539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:49.299524    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44" event={"ID":"181223b0-54c0-4917-ae3c-db275156931c","Type":"ContainerStarted","Data":"ad1eee10120dbf309180b0326d6007560408b2584ab7624ff71e1ff9805371e6"}
Nov 02 17:48:49.305000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:49.304971    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:48:49.340873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:49.338833    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44" podStartSLOduration=2.338813841 podStartE2EDuration="2.338813841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:49.338193715 +0000 UTC m=+941.917603101" watchObservedRunningTime="2026-11-02 17:48:49.338813841 +0000 UTC m=+941.918223228"
Nov 02 17:48:51.080895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.080852    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:48:51.159151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159121    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159175    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159252    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159289    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159349    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159377    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159402    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:51.159581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159435    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.159581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159552    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:51.159581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159570    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:51.159885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.159832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:51.160109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.160087    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.160364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.160115    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.160364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.160132    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.160364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.160147    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.162189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.162155    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:51.162300 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.162196    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "kube-api-access-lxp5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:51.163392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.163367    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:51.261054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261023    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.261254 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261168    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert\") pod \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\" (UID: \"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda\") "
Nov 02 17:48:51.261604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261574    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:51.261942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261923    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.261997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261944    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.261997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261954    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxp5c\" (UniqueName: \"kubernetes.io/projected/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-kube-api-access-lxp5c\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.261997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.261963    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.263827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.263804    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo" pod "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" (UID: "2cf39e49-e7f0-4e3c-bbc7-781a41f12dda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:48:51.357855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.357774    5042 generic.go:356] "Generic (PLEG): container finished" podID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" containerID="3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d" exitCode=0
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.357865    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.357865    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" event={"ID":"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda","Type":"ContainerDied","Data":"3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d"}
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.357988    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd" event={"ID":"2cf39e49-e7f0-4e3c-bbc7-781a41f12dda","Type":"ContainerDied","Data":"b47ab5e6ed3e1560c9977f16b5a44f800faa9b0890fa0a06050e1450e78c061e"}
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.358020    5042 scope.go:122] "RemoveContainer" containerID="3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d"
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.363052    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.370668    5042 scope.go:122] "RemoveContainer" containerID="3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d"
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:51.371078    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d\": container with ID starting with 3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d not found: ID does not exist" containerID="3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d"
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.371106    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d"} err="failed to get container status \"3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d\": rpc error: code = NotFound desc = could not find container \"3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d\": container with ID starting with 3b74c72776c4c4a2a8755ac5ebad096d33e6c54af074ef9aa8459c7f650b8e5d not found: ID does not exist"
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.387442    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"]
Nov 02 17:48:51.403629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.392391    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-g7mdd"]
Nov 02 17:48:51.545922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:51.545892    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" path="/var/lib/kubelet/pods/2cf39e49-e7f0-4e3c-bbc7-781a41f12dda/volumes"
Nov 02 17:48:52.368639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:52.368607    5042 generic.go:356] "Generic (PLEG): container finished" podID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerID="0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78" exitCode=0
Nov 02 17:48:52.369132 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:52.368680    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" event={"ID":"e0ae370c-b74a-4501-828e-414e0e8471d1","Type":"ContainerDied","Data":"0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"}
Nov 02 17:48:52.369260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:52.369243    5042 scope.go:122] "RemoveContainer" containerID="0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"
Nov 02 17:48:52.370897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:52.370872    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:52.370976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:52.370936    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:52.370976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:52.370969    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:53.491863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:53.489410    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" event={"ID":"e0ae370c-b74a-4501-828e-414e0e8471d1","Type":"ContainerStarted","Data":"36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1"}
Nov 02 17:48:54.122865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:54.118773    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:54.122865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:54.119489    5042 patch_prober.go:29] interesting pod/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 17:48:54.122865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:54.119542    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 17:48:55.121904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:55.119943    5042 patch_prober.go:29] interesting pod/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 17:48:55.121904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:55.120006    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 17:48:56.122160 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.122114    5042 patch_prober.go:29] interesting pod/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 17:48:56.122760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.122710    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 17:48:56.355276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.355250    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:56.437213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437182    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437280    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437309    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437339    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437443    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437480    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437541    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437600    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:56.437811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437745    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437797    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy\") pod \"a251a508-27db-49b1-8832-75681664850e\" (UID: \"a251a508-27db-49b1-8832-75681664850e\") "
Nov 02 17:48:56.437930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437880    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:56.437930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.437909    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:56.438151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.438130    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:56.438441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.438418    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.438441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.438442    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.438570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.438456    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.438570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.438470    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a251a508-27db-49b1-8832-75681664850e-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.439174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.439146    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:56.440443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.440422    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:56.440684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.440660    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:48:56.441282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.441261    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "kube-api-access-gb85w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:56.441945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.441909    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo" pod "a251a508-27db-49b1-8832-75681664850e" (UID: "a251a508-27db-49b1-8832-75681664850e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:48:56.540193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.540160    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a251a508-27db-49b1-8832-75681664850e-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.540193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.540193    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.540454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.540208    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb85w\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-kube-api-access-gb85w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.540454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.540223    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a251a508-27db-49b1-8832-75681664850e-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.540454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.540236    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a251a508-27db-49b1-8832-75681664850e-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:48:56.544648 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.544622    5042 generic.go:356] "Generic (PLEG): container finished" podID="a251a508-27db-49b1-8832-75681664850e" containerID="f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801" exitCode=0
Nov 02 17:48:56.544793 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.544745    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"
Nov 02 17:48:56.544793 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.544769    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" event={"ID":"a251a508-27db-49b1-8832-75681664850e","Type":"ContainerDied","Data":"f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801"}
Nov 02 17:48:56.544960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.544822    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp" event={"ID":"a251a508-27db-49b1-8832-75681664850e","Type":"ContainerDied","Data":"dade2c6297afd5a24765c3e580702f9a81f56a33e9d4378ddd9cb776f2a55a19"}
Nov 02 17:48:56.544960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.544875    5042 scope.go:122] "RemoveContainer" containerID="f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801"
Nov 02 17:48:56.558499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.558475    5042 scope.go:122] "RemoveContainer" containerID="f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801"
Nov 02 17:48:56.559032 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:48:56.559010    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801\": container with ID starting with f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801 not found: ID does not exist" containerID="f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801"
Nov 02 17:48:56.559136 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.559041    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801"} err="failed to get container status \"f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801\": rpc error: code = NotFound desc = could not find container \"f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801\": container with ID starting with f20a30a87421f9395b7190d99fdcf5086309d633e926384b31f301d21a621801 not found: ID does not exist"
Nov 02 17:48:56.598003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.597900    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"]
Nov 02 17:48:56.604095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:56.604056    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a976bc-openshift-default-666f9d9749-j2cvp"]
Nov 02 17:48:57.118140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:57.118094    5042 patch_prober.go:29] interesting pod/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 17:48:57.118320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:57.118180    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 17:48:57.568130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:57.568077    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a251a508-27db-49b1-8832-75681664850e" path="/var/lib/kubelet/pods/a251a508-27db-49b1-8832-75681664850e/volumes"
Nov 02 17:48:58.118481 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:58.118437    5042 patch_prober.go:29] interesting pod/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 17:48:58.118665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:58.118519    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 17:48:58.417802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:58.417707    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"]
Nov 02 17:48:58.438701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:58.437941    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-da5f07e6-q5nzk"]
Nov 02 17:48:59.552865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:59.552676    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500aef43-d22f-40d2-a5ba-9f840d722a98" path="/var/lib/kubelet/pods/500aef43-d22f-40d2-a5ba-9f840d722a98/volumes"
Nov 02 17:48:59.712654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:59.712124    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:59.713512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:59.713485    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:48:59.714500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:48:59.714477    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:49:01.362004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:01.361974    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:49:02.696563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:02.696509    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"]
Nov 02 17:49:02.697027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:02.696791    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy" containerID="cri-o://36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1" gracePeriod=30
Nov 02 17:49:04.358117 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358076    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"]
Nov 02 17:49:04.358740 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358723    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" containerName="istio-proxy"
Nov 02 17:49:04.358804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358741    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" containerName="istio-proxy"
Nov 02 17:49:04.358804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358754    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy"
Nov 02 17:49:04.358804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358759    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy"
Nov 02 17:49:04.359011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358871    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf39e49-e7f0-4e3c-bbc7-781a41f12dda" containerName="istio-proxy"
Nov 02 17:49:04.359011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.358885    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a251a508-27db-49b1-8832-75681664850e" containerName="istio-proxy"
Nov 02 17:49:04.364144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.364123    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.375755 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.375725    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-82b305-openshift-default-dockercfg-jwchj\""
Nov 02 17:49:04.384265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.384229    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"]
Nov 02 17:49:04.460468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460431    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.460757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460516    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.460757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460681    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.460757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460721    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.461049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460884    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.461049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460914    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.461049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.460950    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.461049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.461030    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.461289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.461055    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gxc\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569115    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569178    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569225    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569268    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569301    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gxc\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569411    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569438    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569583    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.569614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.573861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.570179    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.594322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.594285    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.594619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.594590    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.595882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.595831    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.596493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.596472    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.627974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.617302    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.627974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.622473    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.638365 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.635698    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gxc\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.648917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.640942    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo\") pod \"e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") " pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:04.689921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:04.687064    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:05.102904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.102660    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"]
Nov 02 17:49:05.105769 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:05.105728    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcecca6b1_7fc9_44cc_9bf8_ba70ee467f33.slice/crio-25e6485d20f8389e983fa29e0c1a0b2644d270f6f50c6a0b050d145ef5f1e8f0 WatchSource:0}: Error finding container 25e6485d20f8389e983fa29e0c1a0b2644d270f6f50c6a0b050d145ef5f1e8f0: Status 404 returned error can't find the container with id 25e6485d20f8389e983fa29e0c1a0b2644d270f6f50c6a0b050d145ef5f1e8f0
Nov 02 17:49:05.109109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.109072    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:05.109247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.109144    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:05.109247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.109174    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:05.726918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.726874    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" event={"ID":"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33","Type":"ContainerStarted","Data":"0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"}
Nov 02 17:49:05.727348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.726948    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" event={"ID":"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33","Type":"ContainerStarted","Data":"25e6485d20f8389e983fa29e0c1a0b2644d270f6f50c6a0b050d145ef5f1e8f0"}
Nov 02 17:49:05.751653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:05.751595    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podStartSLOduration=1.751579103 podStartE2EDuration="1.751579103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:05.74649348 +0000 UTC m=+958.325902857" watchObservedRunningTime="2026-11-02 17:49:05.751579103 +0000 UTC m=+958.330988489"
Nov 02 17:49:06.706380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:06.706336    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:06.736317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:06.736288    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:06.749579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:06.749551    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:06.752020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:06.751965    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:08.175049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.175021    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:49:08.290865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.290510    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.290865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.290573    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309272    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309380    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309517    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309553    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309623    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309705    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309777    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.309809    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs\") pod \"e0ae370c-b74a-4501-828e-414e0e8471d1\" (UID: \"e0ae370c-b74a-4501-828e-414e0e8471d1\") "
Nov 02 17:49:08.314916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.310562    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ae370c-b74a-4501-828e-414e0e8471d1-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.362863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.339969    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:08.399023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.368057    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:08.399023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.368474    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:08.399023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.370643    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:08.399023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.375919    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:08.410906 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.406105    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "kube-api-access-2lxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:08.410906 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.408265    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412681    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412714    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412730    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxvp\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-kube-api-access-2lxvp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412743    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412758    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412772    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ae370c-b74a-4501-828e-414e0e8471d1-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.412784    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.415545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.413260    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token" pod "e0ae370c-b74a-4501-828e-414e0e8471d1" (UID: "e0ae370c-b74a-4501-828e-414e0e8471d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:08.514065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.514031    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ae370c-b74a-4501-828e-414e0e8471d1-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:08.769924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.769891    5042 generic.go:356] "Generic (PLEG): container finished" podID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerID="36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1" exitCode=0
Nov 02 17:49:08.770104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.769954    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"
Nov 02 17:49:08.770104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.769971    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" event={"ID":"e0ae370c-b74a-4501-828e-414e0e8471d1","Type":"ContainerDied","Data":"36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1"}
Nov 02 17:49:08.770104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.770030    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg" event={"ID":"e0ae370c-b74a-4501-828e-414e0e8471d1","Type":"ContainerDied","Data":"9432d913081313e280b9e2d5433e72432ace9740412f95e34421d5e300240a0d"}
Nov 02 17:49:08.770104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.770063    5042 scope.go:122] "RemoveContainer" containerID="36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1"
Nov 02 17:49:08.784180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.784157    5042 scope.go:122] "RemoveContainer" containerID="0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"
Nov 02 17:49:08.797733 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.797704    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"]
Nov 02 17:49:08.802081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.801974    5042 scope.go:122] "RemoveContainer" containerID="36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1"
Nov 02 17:49:08.802402 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:49:08.802377    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1\": container with ID starting with 36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1 not found: ID does not exist" containerID="36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1"
Nov 02 17:49:08.802507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.802417    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1"} err="failed to get container status \"36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1\": rpc error: code = NotFound desc = could not find container \"36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1\": container with ID starting with 36d7e481e51747d2b0fe79b930fa6857f37d44ef6daf0930975da0dd28fc60b1 not found: ID does not exist"
Nov 02 17:49:08.802507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.802443    5042 scope.go:122] "RemoveContainer" containerID="0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"
Nov 02 17:49:08.802778 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:49:08.802759    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78\": container with ID starting with 0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78 not found: ID does not exist" containerID="0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"
Nov 02 17:49:08.802861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.802786    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78"} err="failed to get container status \"0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78\": rpc error: code = NotFound desc = could not find container \"0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78\": container with ID starting with 0ccce47722b4e21b41403226ac8f88c9626b4029b87f793167bd709b92523b78 not found: ID does not exist"
Nov 02 17:49:08.805161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.805138    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-11f0a3-openshift-default-74d55d7b87-zpplg"]
Nov 02 17:49:08.855697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.855669    5042 scope.go:122] "RemoveContainer" containerID="b54ae4dc3b0a421d017224906a677b15481f6924d31930bc24263603401d5d0c"
Nov 02 17:49:08.868228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.868003    5042 scope.go:122] "RemoveContainer" containerID="50d9a17b59244761e7f141792152e359334439b9513a57a0ca8b2a9d59b012f0"
Nov 02 17:49:08.881237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.881214    5042 scope.go:122] "RemoveContainer" containerID="a3cfe1b929404a2da2340e1499795e35fbecf6d3827fe66fbd586f74a658b623"
Nov 02 17:49:08.894037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:08.894011    5042 scope.go:122] "RemoveContainer" containerID="ce7c0debac81e357f9aa26d2cd501773f1f36d4524024cfa4b1590866124d20a"
Nov 02 17:49:09.544118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.544087    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" path="/var/lib/kubelet/pods/e0ae370c-b74a-4501-828e-414e0e8471d1/volumes"
Nov 02 17:49:09.711105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.711072    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"]
Nov 02 17:49:09.711738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.711723    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy"
Nov 02 17:49:09.711738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.711739    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy"
Nov 02 17:49:09.711855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.711763    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy"
Nov 02 17:49:09.711855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.711769    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy"
Nov 02 17:49:09.711922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.711875    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy"
Nov 02 17:49:09.717402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.717375    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:09.721789 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.721765    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-82b305\""
Nov 02 17:49:09.726109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.726087    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"]
Nov 02 17:49:09.783696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.783662    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhxf\" (UniqueName: \"kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf\") pod \"payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:09.783940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.783744    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:09.895927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.893517    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhxf\" (UniqueName: \"kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf\") pod \"payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:09.895927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.893616    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:09.899870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.896388    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:09.926887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:09.923674    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhxf\" (UniqueName: \"kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf\") pod \"payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:10.034475 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.034436    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:10.380013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.379975    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"]
Nov 02 17:49:10.385145 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:10.385049    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758e32a7_6861_4ae2_a743_e4d9a55a2f20.slice/crio-76fa97d3a808cc4325c87abfc1b9c11cb2b80bd5ee0b87a255fc6e985947402d WatchSource:0}: Error finding container 76fa97d3a808cc4325c87abfc1b9c11cb2b80bd5ee0b87a255fc6e985947402d: Status 404 returned error can't find the container with id 76fa97d3a808cc4325c87abfc1b9c11cb2b80bd5ee0b87a255fc6e985947402d
Nov 02 17:49:10.756185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.756148    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"]
Nov 02 17:49:10.804446 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.804411    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h" event={"ID":"758e32a7-6861-4ae2-a743-e4d9a55a2f20","Type":"ContainerStarted","Data":"ef8e553a163ee9147af9a3c1379d7f60b10a8ca799c63f53316ceb9f16a1269e"}
Nov 02 17:49:10.804709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.804691    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h" event={"ID":"758e32a7-6861-4ae2-a743-e4d9a55a2f20","Type":"ContainerStarted","Data":"76fa97d3a808cc4325c87abfc1b9c11cb2b80bd5ee0b87a255fc6e985947402d"}
Nov 02 17:49:10.804822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.804811    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:10.843156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:10.843087    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h" podStartSLOduration=1.843066834 podStartE2EDuration="1.843066834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:10.838057468 +0000 UTC m=+963.417466867" watchObservedRunningTime="2026-11-02 17:49:10.843066834 +0000 UTC m=+963.422476220"
Nov 02 17:49:15.595576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.595537    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"]
Nov 02 17:49:15.598861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.598813    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ae370c-b74a-4501-828e-414e0e8471d1" containerName="istio-proxy"
Nov 02 17:49:15.609348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.605995    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:15.637317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.635283    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4c9298-dockercfg-92v4d\""
Nov 02 17:49:15.642922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.642587    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69j7\" (UniqueName: \"kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:15.642922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.642697    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:15.656480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.656351    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"]
Nov 02 17:49:15.661496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.661450    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"]
Nov 02 17:49:15.668509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.668474    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"]
Nov 02 17:49:15.668686 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.668603    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:15.843175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.843102    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44" podUID="181223b0-54c0-4917-ae3c-db275156931c" containerName="payload-processing" containerID="cri-o://ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b" gracePeriod=25
Nov 02 17:49:15.975252 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.857633    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69j7\" (UniqueName: \"kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:15.975557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.975530    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:15.975701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.950339    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69j7\" (UniqueName: \"kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:15.994270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.975911    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:15.994270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.991141    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfdv\" (UniqueName: \"kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv\") pod \"payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:15.994270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:15.977192    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:16.103893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.095777    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfdv\" (UniqueName: \"kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv\") pod \"payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:16.103893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.096080    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:16.103893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.099292    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:16.190895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.184897    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfdv\" (UniqueName: \"kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv\") pod \"payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:16.243431 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.241995    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:16.298995 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.286097    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:16.541233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.541001    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:49:16.640265 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.640146    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn\" (UniqueName: \"kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn\") pod \"181223b0-54c0-4917-ae3c-db275156931c\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") "
Nov 02 17:49:16.641201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.641080    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume\") pod \"181223b0-54c0-4917-ae3c-db275156931c\" (UID: \"181223b0-54c0-4917-ae3c-db275156931c\") "
Nov 02 17:49:16.641559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.641531    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume" pod "181223b0-54c0-4917-ae3c-db275156931c" (UID: "181223b0-54c0-4917-ae3c-db275156931c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:16.642068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.642042    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181223b0-54c0-4917-ae3c-db275156931c-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:16.649284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.649242    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn" pod "181223b0-54c0-4917-ae3c-db275156931c" (UID: "181223b0-54c0-4917-ae3c-db275156931c"). InnerVolumeSpecName "kube-api-access-m74mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:16.678991 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.678565    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"]
Nov 02 17:49:16.763062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.753771    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m74mn\" (UniqueName: \"kubernetes.io/projected/181223b0-54c0-4917-ae3c-db275156931c-kube-api-access-m74mn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:16.858178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.858066    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"]
Nov 02 17:49:16.990865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.990635    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5" event={"ID":"98e1b231-b167-4dc3-8820-0126f6a8d008","Type":"ContainerStarted","Data":"c3e0315e247247857514713ceca54827062b54001b1eab7508edb369c8b041a4"}
Nov 02 17:49:16.994861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.993887    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn" event={"ID":"7d0066fa-1b96-4961-a57b-b616ef3f4dab","Type":"ContainerStarted","Data":"b15abc27adbdb8c10ca93bc65f0cf2fba06fae7110c234b1eab2354204d4eab7"}
Nov 02 17:49:16.996891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.995605    5042 generic.go:356] "Generic (PLEG): container finished" podID="181223b0-54c0-4917-ae3c-db275156931c" containerID="ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b" exitCode=0
Nov 02 17:49:16.996891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.995709    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"
Nov 02 17:49:16.996891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.995740    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44" event={"ID":"181223b0-54c0-4917-ae3c-db275156931c","Type":"ContainerDied","Data":"ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b"}
Nov 02 17:49:16.996891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.995792    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44" event={"ID":"181223b0-54c0-4917-ae3c-db275156931c","Type":"ContainerDied","Data":"ad1eee10120dbf309180b0326d6007560408b2584ab7624ff71e1ff9805371e6"}
Nov 02 17:49:16.996891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:16.995823    5042 scope.go:122] "RemoveContainer" containerID="ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b"
Nov 02 17:49:17.009784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.009754    5042 scope.go:122] "RemoveContainer" containerID="ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b"
Nov 02 17:49:17.010215 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:49:17.010195    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b\": container with ID starting with ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b not found: ID does not exist" containerID="ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b"
Nov 02 17:49:17.010297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.010229    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b"} err="failed to get container status \"ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b\": rpc error: code = NotFound desc = could not find container \"ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b\": container with ID starting with ca67364b1364b673462734e689ac4f461932eaa2a9327b0f229ab896341c7c1b not found: ID does not exist"
Nov 02 17:49:17.027318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.027284    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"]
Nov 02 17:49:17.032372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.032341    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-86597cb9bc-jrw44"]
Nov 02 17:49:17.601220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.601151    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181223b0-54c0-4917-ae3c-db275156931c" path="/var/lib/kubelet/pods/181223b0-54c0-4917-ae3c-db275156931c/volumes"
Nov 02 17:49:17.745724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.745675    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"]
Nov 02 17:49:17.747649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.747356    5042 cpu_manager.go:414] "containerMap: removing container" podUID="181223b0-54c0-4917-ae3c-db275156931c" containerName="payload-processing"
Nov 02 17:49:17.759522 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.759487    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181223b0-54c0-4917-ae3c-db275156931c" containerName="payload-processing"
Nov 02 17:49:17.762049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.760159    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="181223b0-54c0-4917-ae3c-db275156931c" containerName="payload-processing"
Nov 02 17:49:17.766329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.766284    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"]
Nov 02 17:49:17.766550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.766415    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:17.771666 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.771526    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d1fc77\""
Nov 02 17:49:17.772405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.771892    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d1fc77-dockercfg-chvwr\""
Nov 02 17:49:17.811679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.811642    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"]
Nov 02 17:49:17.823452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.823422    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:17.830267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.830194    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"]
Nov 02 17:49:17.908527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.908148    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l2k5\" (UniqueName: \"kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5\") pod \"payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:17.908527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:17.908256    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:18.020748 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.020716    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfln\" (UniqueName: \"kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.025652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.025623    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.025963 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.025944    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l2k5\" (UniqueName: \"kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5\") pod \"payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:18.066054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.066010    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:18.067726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.067680    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:18.070963 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.070935    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l2k5\" (UniqueName: \"kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5\") pod \"payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:18.090515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.090474    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:18.178788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.178751    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfln\" (UniqueName: \"kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.179006 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.178808    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.181137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.180195    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.218972 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.218940    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfln\" (UniqueName: \"kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln\") pod \"payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.321517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.321189    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:18.424779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.424737    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"]
Nov 02 17:49:18.425742 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:18.425713    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode586ed01_04be_43cf_bb26_78f6bf030ea8.slice/crio-4524a0c859ea9315321b831ef8961d23b695c4dedc9d734c151a56b8274ed2a9 WatchSource:0}: Error finding container 4524a0c859ea9315321b831ef8961d23b695c4dedc9d734c151a56b8274ed2a9: Status 404 returned error can't find the container with id 4524a0c859ea9315321b831ef8961d23b695c4dedc9d734c151a56b8274ed2a9
Nov 02 17:49:18.579104 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:49:18.579054    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:49:18.698717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:18.698672    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"]
Nov 02 17:49:18.708656 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:18.708617    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c6229a_5c06_4fd4_a8c5_e6ee7d3cd506.slice/crio-812a70c7ce82ef88235b637786eb40c6bacf6caa343567beb67efcb7c48ca8a8 WatchSource:0}: Error finding container 812a70c7ce82ef88235b637786eb40c6bacf6caa343567beb67efcb7c48ca8a8: Status 404 returned error can't find the container with id 812a70c7ce82ef88235b637786eb40c6bacf6caa343567beb67efcb7c48ca8a8
Nov 02 17:49:19.109069 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:19.108968    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb" event={"ID":"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506","Type":"ContainerStarted","Data":"812a70c7ce82ef88235b637786eb40c6bacf6caa343567beb67efcb7c48ca8a8"}
Nov 02 17:49:19.150473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:19.150365    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv" event={"ID":"e586ed01-04be-43cf-bb26-78f6bf030ea8","Type":"ContainerStarted","Data":"4524a0c859ea9315321b831ef8961d23b695c4dedc9d734c151a56b8274ed2a9"}
Nov 02 17:49:21.195381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.195278    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn" event={"ID":"7d0066fa-1b96-4961-a57b-b616ef3f4dab","Type":"ContainerStarted","Data":"bbd1b5ac8a020ca238b088fffb1849bc61a2e3d20d6a100d4afe78f4d4b734c0"}
Nov 02 17:49:21.196517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.196450    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:21.198689 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.198652    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5" event={"ID":"98e1b231-b167-4dc3-8820-0126f6a8d008","Type":"ContainerStarted","Data":"50e62e089568b27e182b0356fc701e552e73fba423dc1b867d481b8b8bfccbbc"}
Nov 02 17:49:21.199538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.199480    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:21.236604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.236543    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn" podStartSLOduration=1.9072296180000001 podStartE2EDuration="6.236523749s" totalImagesPullingTime="4.329294131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:21.236420422 +0000 UTC m=+973.815829836" watchObservedRunningTime="2026-11-02 17:49:21.236523749 +0000 UTC m=+973.815933135"
Nov 02 17:49:21.262264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.261721    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5" podStartSLOduration=2.055045054 podStartE2EDuration="6.261704599s" totalImagesPullingTime="4.206659545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:21.256234358 +0000 UTC m=+973.835643744" watchObservedRunningTime="2026-11-02 17:49:21.261704599 +0000 UTC m=+973.841113985"
Nov 02 17:49:21.820967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:21.820937    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:22.303864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:22.302824    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb" event={"ID":"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506","Type":"ContainerStarted","Data":"608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087"}
Nov 02 17:49:22.304361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:22.304300    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:22.308370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:22.308335    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv" event={"ID":"e586ed01-04be-43cf-bb26-78f6bf030ea8","Type":"ContainerStarted","Data":"c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6"}
Nov 02 17:49:22.308537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:22.308397    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:22.331266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:22.327670    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb" podStartSLOduration=2.904226616 podStartE2EDuration="5.327649131s" totalImagesPullingTime="2.423422515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:22.323980604 +0000 UTC m=+974.903390007" watchObservedRunningTime="2026-11-02 17:49:22.327649131 +0000 UTC m=+974.907058518"
Nov 02 17:49:22.347140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:22.347072    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv" podStartSLOduration=2.620849942 podStartE2EDuration="5.347050584s" totalImagesPullingTime="2.726200642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:22.346348248 +0000 UTC m=+974.925757635" watchObservedRunningTime="2026-11-02 17:49:22.347050584 +0000 UTC m=+974.926459973"
Nov 02 17:49:28.110869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:28.104737    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"]
Nov 02 17:49:31.027360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:31.027316    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"]
Nov 02 17:49:31.031902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:31.031869    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-823c0021-rt9pg"]
Nov 02 17:49:31.545516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:31.545479    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c40dfa-984e-4eed-b3d6-4399a6b29887" path="/var/lib/kubelet/pods/83c40dfa-984e-4eed-b3d6-4399a6b29887/volumes"
Nov 02 17:49:33.106015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.105907    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h" podUID="758e32a7-6861-4ae2-a743-e4d9a55a2f20" containerName="payload-processing" containerID="cri-o://ef8e553a163ee9147af9a3c1379d7f60b10a8ca799c63f53316ceb9f16a1269e" gracePeriod=25
Nov 02 17:49:33.343547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.343473    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:49:33.344741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.344721    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:49:33.347938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.347914    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:49:33.453332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.453292    5042 generic.go:356] "Generic (PLEG): container finished" podID="758e32a7-6861-4ae2-a743-e4d9a55a2f20" containerID="ef8e553a163ee9147af9a3c1379d7f60b10a8ca799c63f53316ceb9f16a1269e" exitCode=0
Nov 02 17:49:33.453513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.453347    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h" event={"ID":"758e32a7-6861-4ae2-a743-e4d9a55a2f20","Type":"ContainerDied","Data":"ef8e553a163ee9147af9a3c1379d7f60b10a8ca799c63f53316ceb9f16a1269e"}
Nov 02 17:49:33.500868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.476052    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:33.567307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.567270    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume\") pod \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") "
Nov 02 17:49:33.567483 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.567448    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf\" (UniqueName: \"kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf\") pod \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\" (UID: \"758e32a7-6861-4ae2-a743-e4d9a55a2f20\") "
Nov 02 17:49:33.569346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.569312    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume" pod "758e32a7-6861-4ae2-a743-e4d9a55a2f20" (UID: "758e32a7-6861-4ae2-a743-e4d9a55a2f20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:33.588048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.585779    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf" pod "758e32a7-6861-4ae2-a743-e4d9a55a2f20" (UID: "758e32a7-6861-4ae2-a743-e4d9a55a2f20"). InnerVolumeSpecName "kube-api-access-dqhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:33.627870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.623946    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"]
Nov 02 17:49:33.627870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.625088    5042 cpu_manager.go:414] "containerMap: removing container" podUID="758e32a7-6861-4ae2-a743-e4d9a55a2f20" containerName="payload-processing"
Nov 02 17:49:33.627870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.625111    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758e32a7-6861-4ae2-a743-e4d9a55a2f20" containerName="payload-processing"
Nov 02 17:49:33.627870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.625294    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="758e32a7-6861-4ae2-a743-e4d9a55a2f20" containerName="payload-processing"
Nov 02 17:49:33.632187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.631906    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:33.634062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.634031    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"]
Nov 02 17:49:33.667730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.649507    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-82b305-dockercfg-bc26l\""
Nov 02 17:49:33.756923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.722017    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:33.756923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.722295    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vqr\" (UniqueName: \"kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr\") pod \"payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:33.756923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.722555    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/758e32a7-6861-4ae2-a743-e4d9a55a2f20-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:33.756923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.722575    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhxf\" (UniqueName: \"kubernetes.io/projected/758e32a7-6861-4ae2-a743-e4d9a55a2f20-kube-api-access-dqhxf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:49:33.893867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.829130    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vqr\" (UniqueName: \"kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr\") pod \"payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:33.924855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.899157    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:33.925183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.900710    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:33.976862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:33.972576    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vqr\" (UniqueName: \"kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr\") pod \"payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:34.085314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.085213    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:34.350171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.350080    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:49:34.467235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.467188    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"]
Nov 02 17:49:34.491388 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:34.476200    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7d3815_1deb_46a6_92be_dfed9a7139a0.slice/crio-5ddbcd5f8d8bedce0b2a6f90e3ce0f027d275aac99ee93bd9604d9d45a26cfe0 WatchSource:0}: Error finding container 5ddbcd5f8d8bedce0b2a6f90e3ce0f027d275aac99ee93bd9604d9d45a26cfe0: Status 404 returned error can't find the container with id 5ddbcd5f8d8bedce0b2a6f90e3ce0f027d275aac99ee93bd9604d9d45a26cfe0
Nov 02 17:49:34.524175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.524141    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h" event={"ID":"758e32a7-6861-4ae2-a743-e4d9a55a2f20","Type":"ContainerDied","Data":"76fa97d3a808cc4325c87abfc1b9c11cb2b80bd5ee0b87a255fc6e985947402d"}
Nov 02 17:49:34.524327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.524212    5042 scope.go:122] "RemoveContainer" containerID="ef8e553a163ee9147af9a3c1379d7f60b10a8ca799c63f53316ceb9f16a1269e"
Nov 02 17:49:34.524934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.524911    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"
Nov 02 17:49:34.559018 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.555910    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"]
Nov 02 17:49:34.561472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:34.561447    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-82b305-799459bc57-lsd7h"]
Nov 02 17:49:35.546184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:35.546136    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758e32a7-6861-4ae2-a743-e4d9a55a2f20" path="/var/lib/kubelet/pods/758e32a7-6861-4ae2-a743-e4d9a55a2f20/volumes"
Nov 02 17:49:35.546716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:35.546495    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f" event={"ID":"7c7d3815-1deb-46a6-92be-dfed9a7139a0","Type":"ContainerStarted","Data":"b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5"}
Nov 02 17:49:35.546716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:35.546548    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f" event={"ID":"7c7d3815-1deb-46a6-92be-dfed9a7139a0","Type":"ContainerStarted","Data":"5ddbcd5f8d8bedce0b2a6f90e3ce0f027d275aac99ee93bd9604d9d45a26cfe0"}
Nov 02 17:49:35.546716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:35.546571    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:35.558970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:35.558910    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f" podStartSLOduration=2.558890484 podStartE2EDuration="2.558890484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:35.555928825 +0000 UTC m=+988.135338212" watchObservedRunningTime="2026-11-02 17:49:35.558890484 +0000 UTC m=+988.138299870"
Nov 02 17:49:37.794680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:37.793965    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:49:37.794680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:37.794060    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:40.628027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:40.625562    5042 generic.go:356] "Generic (PLEG): container finished" podID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerID="0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8" exitCode=0
Nov 02 17:49:40.628027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:40.625795    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" event={"ID":"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33","Type":"ContainerDied","Data":"0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"}
Nov 02 17:49:40.628027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:40.626411    5042 scope.go:122] "RemoveContainer" containerID="0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"
Nov 02 17:49:40.629167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:40.628491    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:40.629167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:40.628577    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:40.629167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:40.628625    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:41.662331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:41.662293    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" event={"ID":"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33","Type":"ContainerStarted","Data":"78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61"}
Nov 02 17:49:41.701871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:41.696222    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:41.701871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:41.696348    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:49:41.701871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:41.696411    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 17:49:42.695259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:42.695223    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:49:42.695880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:42.695824    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 17:49:43.046706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.046610    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"]
Nov 02 17:49:43.054186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.054145    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a140a353-89qv5"]
Nov 02 17:49:43.130810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.130776    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"]
Nov 02 17:49:43.136950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.136392    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.147323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.147214    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"]
Nov 02 17:49:43.234868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.206139    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f7957\""
Nov 02 17:49:43.234868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.206305    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f7957-dockercfg-lb8fn\""
Nov 02 17:49:43.324924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.282987    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwqw\" (UniqueName: \"kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw\") pod \"payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.324924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.305688    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.412826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.412785    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwqw\" (UniqueName: \"kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw\") pod \"payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.422184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.422141    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.439870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.423549    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.552458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.552428    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d3d523-9fb1-443c-9ebb-aa3f592c20bf" path="/var/lib/kubelet/pods/80d3d523-9fb1-443c-9ebb-aa3f592c20bf/volumes"
Nov 02 17:49:43.557755 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.557724    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwqw\" (UniqueName: \"kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw\") pod \"payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:43.693961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.693320    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:49:43.693961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.693401    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 17:49:43.756022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:43.755988    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:44.132325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.131126    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"]
Nov 02 17:49:44.143221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.143190    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.154655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.154615    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"]
Nov 02 17:49:44.156056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.156031    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dd1cba-dockercfg-4p7xq\""
Nov 02 17:49:44.157791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.157763    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dd1cba\""
Nov 02 17:49:44.164429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.164168    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.164429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.164250    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8cn\" (UniqueName: \"kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.166801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.166607    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"]
Nov 02 17:49:44.184874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.184081    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.364988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.314696    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8cn\" (UniqueName: \"kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.364988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.314761    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.364988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.314806    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66tp\" (UniqueName: \"kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp\") pod \"payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.488390 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:44.488276    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93435d3_2314_4f81_ac77_5edfd7cefa67.slice/crio-184fdfa8963780a7e4696f942f91d94169d7ee41057610a73887ea46f66d6404 WatchSource:0}: Error finding container 184fdfa8963780a7e4696f942f91d94169d7ee41057610a73887ea46f66d6404: Status 404 returned error can't find the container with id 184fdfa8963780a7e4696f942f91d94169d7ee41057610a73887ea46f66d6404
Nov 02 17:49:44.510914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.488881    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.510914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.491325    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.510914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.508494    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8cn\" (UniqueName: \"kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn\") pod \"payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.543105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.541570    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"]
Nov 02 17:49:44.570557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.570523    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"]
Nov 02 17:49:44.591973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.591938    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.592157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.592014    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66tp\" (UniqueName: \"kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp\") pod \"payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.594107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.594042    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.608449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.608417    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66tp\" (UniqueName: \"kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp\") pod \"payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:44.722864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.720475    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:49:44.722864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.720531    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 17:49:44.745367 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.745333    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln" event={"ID":"c93435d3-2314-4f81-ac77-5edfd7cefa67","Type":"ContainerStarted","Data":"e95aef39ccfbcf3a8480ef4de87b12dbb185129a1538b610064c3577d2044121"}
Nov 02 17:49:44.745560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.745394    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln" event={"ID":"c93435d3-2314-4f81-ac77-5edfd7cefa67","Type":"ContainerStarted","Data":"184fdfa8963780a7e4696f942f91d94169d7ee41057610a73887ea46f66d6404"}
Nov 02 17:49:44.745982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.745963    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:44.770354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.770256    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln" podStartSLOduration=1.770241457 podStartE2EDuration="1.770241457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:44.765627795 +0000 UTC m=+997.345037181" watchObservedRunningTime="2026-11-02 17:49:44.770241457 +0000 UTC m=+997.349650843"
Nov 02 17:49:44.805079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.804705    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:44.966917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:44.966779    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:45.208881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.205706    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"]
Nov 02 17:49:45.336182 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:45.336152    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9b6e90_ee95_4f05_85db_6fef1576215b.slice/crio-8339dfa99075b1459efd48728804ea566e12c7d093d0d486460077eb368ef3c2 WatchSource:0}: Error finding container 8339dfa99075b1459efd48728804ea566e12c7d093d0d486460077eb368ef3c2: Status 404 returned error can't find the container with id 8339dfa99075b1459efd48728804ea566e12c7d093d0d486460077eb368ef3c2
Nov 02 17:49:45.337033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.337001    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"]
Nov 02 17:49:45.709884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.706311    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:49:45.709884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.706375    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 17:49:45.757335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.757293    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc" event={"ID":"5a4e0482-cbbb-4eb9-973d-2774368f8c89","Type":"ContainerStarted","Data":"081a67310e01db93573cc86abebc35635ad0ab35bb9e25615267642cf60e64f5"}
Nov 02 17:49:45.757540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.757367    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:45.757540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.757392    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc" event={"ID":"5a4e0482-cbbb-4eb9-973d-2774368f8c89","Type":"ContainerStarted","Data":"5e3b246854ccb38313111e43baeea57c6e7a56c84d177b37dd848589ea9b3e76"}
Nov 02 17:49:45.759446 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.759420    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz" event={"ID":"9c9b6e90-ee95-4f05-85db-6fef1576215b","Type":"ContainerStarted","Data":"c8d5fe16988ad84ce3cf955994c231ec2c7ca69556729bf50168cac6a2e2996b"}
Nov 02 17:49:45.759581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.759465    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz" event={"ID":"9c9b6e90-ee95-4f05-85db-6fef1576215b","Type":"ContainerStarted","Data":"8339dfa99075b1459efd48728804ea566e12c7d093d0d486460077eb368ef3c2"}
Nov 02 17:49:45.759581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.759560    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:45.808524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.808473    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc" podStartSLOduration=1.808457041 podStartE2EDuration="1.808457041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:45.779342388 +0000 UTC m=+998.358751777" watchObservedRunningTime="2026-11-02 17:49:45.808457041 +0000 UTC m=+998.387866428"
Nov 02 17:49:45.810294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:45.810254    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz" podStartSLOduration=1.8102433690000002 podStartE2EDuration="1.810243369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:45.808356319 +0000 UTC m=+998.387765707" watchObservedRunningTime="2026-11-02 17:49:45.810243369 +0000 UTC m=+998.389652755"
Nov 02 17:49:46.570869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:46.569011    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:49:46.693938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:46.693899    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:49:46.700869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:46.694199    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 17:49:47.697166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:47.697130    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:47.697678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:47.697547    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:47.698449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:47.698427    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:49:48.543094 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:49:48.543052    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:49:55.031178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.031131    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"]
Nov 02 17:49:55.035583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.035556    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0d6de8-dfd5x"]
Nov 02 17:49:55.408624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.408530    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"]
Nov 02 17:49:55.414512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.414486    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.424424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.424396    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c8832b-openshift-default-dockercfg-2sk8m\""
Nov 02 17:49:55.451548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.444765    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"]
Nov 02 17:49:55.547730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.547695    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e1fbbd-28ff-4fe3-bbd3-871b37665b01" path="/var/lib/kubelet/pods/78e1fbbd-28ff-4fe3-bbd3-871b37665b01/volumes"
Nov 02 17:49:55.569325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569292    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569347    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569443    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569528    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569556    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96h9c\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569575    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569600    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569648    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.569787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.569709    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.675917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.675176    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.675917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.675237    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96h9c\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.675917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.675268    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.690977    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.691360    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.691479    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.691604    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.691815    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.691899    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.693866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.691957    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.700501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.700255    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.733865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.720579    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.733865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.720859    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.733865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.727019    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.753863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.743802    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.772876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.765327    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.772876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.772342    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:49:55.798872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.784696    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:55.798872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:55.785812    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96h9c\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c\") pod \"e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") " pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:56.056867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.056798    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:56.355638 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:49:56.355602    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15640b9_72f5_4254_a525_e72f7a75a24e.slice/crio-b026ccfae00e5f7375628d68c6dc57198378b9142b9ecbec3f9efff76fc5b739 WatchSource:0}: Error finding container b026ccfae00e5f7375628d68c6dc57198378b9142b9ecbec3f9efff76fc5b739: Status 404 returned error can't find the container with id b026ccfae00e5f7375628d68c6dc57198378b9142b9ecbec3f9efff76fc5b739
Nov 02 17:49:56.356119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.356093    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"]
Nov 02 17:49:56.358625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.358593    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:56.358709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.358661    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:56.358709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.358692    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:56.771801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.771776    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:49:56.772019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.771923    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:49:56.976920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.976872    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" event={"ID":"c15640b9-72f5-4254-a525-e72f7a75a24e","Type":"ContainerStarted","Data":"9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"}
Nov 02 17:49:56.977110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.976939    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" event={"ID":"c15640b9-72f5-4254-a525-e72f7a75a24e","Type":"ContainerStarted","Data":"b026ccfae00e5f7375628d68c6dc57198378b9142b9ecbec3f9efff76fc5b739"}
Nov 02 17:49:56.997419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:56.997338    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podStartSLOduration=1.997318986 podStartE2EDuration="1.997318986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:56.996323011 +0000 UTC m=+1009.575732397" watchObservedRunningTime="2026-11-02 17:49:56.997318986 +0000 UTC m=+1009.576728373"
Nov 02 17:49:57.057485 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:57.057371    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:57.064533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:57.064505    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:57.986452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:57.986407    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:49:57.987305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:49:57.987288    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:50:07.020163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:07.014971    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"]
Nov 02 17:50:07.869407 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:07.869361    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"]
Nov 02 17:50:07.884719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:07.884682    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"]
Nov 02 17:50:09.143379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:09.143349    5042 scope.go:122] "RemoveContainer" containerID="28bbcd0e0852cc14340b74019a73fbc08c3d9ea9ac3c150cc0f5a363b13c5e05"
Nov 02 17:50:09.154783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:09.154759    5042 scope.go:122] "RemoveContainer" containerID="e808f16e7200a783e940489b559250fc64704dfc49e52ac3517c40c5a964da97"
Nov 02 17:50:09.165120 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:09.165085    5042 scope.go:122] "RemoveContainer" containerID="e1663ce34e9c9679e4ed11a503f66d16d398132775f150a13c332ec47afb6f55"
Nov 02 17:50:09.175500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:09.175479    5042 scope.go:122] "RemoveContainer" containerID="d613585d04d202f0d760a78d332c0e9d6290bd9f4c32dfd74403aeae904f29a8"
Nov 02 17:50:11.030352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:11.030303    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"]
Nov 02 17:50:11.035103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:11.035074    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b2ac45e-stjsc"]
Nov 02 17:50:11.546340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:11.546301    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be480d4-5ae6-488b-b883-35455abb3e0b" path="/var/lib/kubelet/pods/6be480d4-5ae6-488b-b883-35455abb3e0b/volumes"
Nov 02 17:50:12.016280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.016215    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln" podUID="c93435d3-2314-4f81-ac77-5edfd7cefa67" containerName="payload-processing" containerID="cri-o://e95aef39ccfbcf3a8480ef4de87b12dbb185129a1538b610064c3577d2044121" gracePeriod=25
Nov 02 17:50:12.164948 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.164914    5042 generic.go:356] "Generic (PLEG): container finished" podID="c93435d3-2314-4f81-ac77-5edfd7cefa67" containerID="e95aef39ccfbcf3a8480ef4de87b12dbb185129a1538b610064c3577d2044121" exitCode=0
Nov 02 17:50:12.165342 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.164959    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln" event={"ID":"c93435d3-2314-4f81-ac77-5edfd7cefa67","Type":"ContainerDied","Data":"e95aef39ccfbcf3a8480ef4de87b12dbb185129a1538b610064c3577d2044121"}
Nov 02 17:50:12.308537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.308509    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:50:12.448868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.448818    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw\" (UniqueName: \"kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw\") pod \"c93435d3-2314-4f81-ac77-5edfd7cefa67\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") "
Nov 02 17:50:12.449069 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.448925    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume\") pod \"c93435d3-2314-4f81-ac77-5edfd7cefa67\" (UID: \"c93435d3-2314-4f81-ac77-5edfd7cefa67\") "
Nov 02 17:50:12.449317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.449294    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume" pod "c93435d3-2314-4f81-ac77-5edfd7cefa67" (UID: "c93435d3-2314-4f81-ac77-5edfd7cefa67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:12.449648 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.449629    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c93435d3-2314-4f81-ac77-5edfd7cefa67-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:12.451165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.451139    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw" pod "c93435d3-2314-4f81-ac77-5edfd7cefa67" (UID: "c93435d3-2314-4f81-ac77-5edfd7cefa67"). InnerVolumeSpecName "kube-api-access-tnwqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:12.551335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.551235    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnwqw\" (UniqueName: \"kubernetes.io/projected/c93435d3-2314-4f81-ac77-5edfd7cefa67-kube-api-access-tnwqw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:12.870824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.869916    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz" podUID="9c9b6e90-ee95-4f05-85db-6fef1576215b" containerName="payload-processing" containerID="cri-o://c8d5fe16988ad84ce3cf955994c231ec2c7ca69556729bf50168cac6a2e2996b" gracePeriod=25
Nov 02 17:50:12.888906 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:12.888855    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc" podUID="5a4e0482-cbbb-4eb9-973d-2774368f8c89" containerName="payload-pre-processing" containerID="cri-o://081a67310e01db93573cc86abebc35635ad0ab35bb9e25615267642cf60e64f5" gracePeriod=25
Nov 02 17:50:13.183644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.183244    5042 generic.go:356] "Generic (PLEG): container finished" podID="9c9b6e90-ee95-4f05-85db-6fef1576215b" containerID="c8d5fe16988ad84ce3cf955994c231ec2c7ca69556729bf50168cac6a2e2996b" exitCode=0
Nov 02 17:50:13.183644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.183518    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz" event={"ID":"9c9b6e90-ee95-4f05-85db-6fef1576215b","Type":"ContainerDied","Data":"c8d5fe16988ad84ce3cf955994c231ec2c7ca69556729bf50168cac6a2e2996b"}
Nov 02 17:50:13.186105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.186071    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln" event={"ID":"c93435d3-2314-4f81-ac77-5edfd7cefa67","Type":"ContainerDied","Data":"184fdfa8963780a7e4696f942f91d94169d7ee41057610a73887ea46f66d6404"}
Nov 02 17:50:13.186256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.186134    5042 scope.go:122] "RemoveContainer" containerID="e95aef39ccfbcf3a8480ef4de87b12dbb185129a1538b610064c3577d2044121"
Nov 02 17:50:13.186322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.186284    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"
Nov 02 17:50:13.193079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.193045    5042 generic.go:356] "Generic (PLEG): container finished" podID="5a4e0482-cbbb-4eb9-973d-2774368f8c89" containerID="081a67310e01db93573cc86abebc35635ad0ab35bb9e25615267642cf60e64f5" exitCode=0
Nov 02 17:50:13.193195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.193163    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc" event={"ID":"5a4e0482-cbbb-4eb9-973d-2774368f8c89","Type":"ContainerDied","Data":"081a67310e01db93573cc86abebc35635ad0ab35bb9e25615267642cf60e64f5"}
Nov 02 17:50:13.241142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.241097    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"]
Nov 02 17:50:13.246789 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.246759    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-6f59c86459-q6xln"]
Nov 02 17:50:13.259084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.259060    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:50:13.312726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.312695    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:50:13.367645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.367249    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume\") pod \"9c9b6e90-ee95-4f05-85db-6fef1576215b\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") "
Nov 02 17:50:13.367875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.367789    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp\" (UniqueName: \"kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp\") pod \"9c9b6e90-ee95-4f05-85db-6fef1576215b\" (UID: \"9c9b6e90-ee95-4f05-85db-6fef1576215b\") "
Nov 02 17:50:13.392631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.371932    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume" pod "9c9b6e90-ee95-4f05-85db-6fef1576215b" (UID: "9c9b6e90-ee95-4f05-85db-6fef1576215b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:13.480877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.474913    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume\") pod \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") "
Nov 02 17:50:13.516476 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.516409    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume" pod "5a4e0482-cbbb-4eb9-973d-2774368f8c89" (UID: "5a4e0482-cbbb-4eb9-973d-2774368f8c89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:13.517342 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.517319    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn\" (UniqueName: \"kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn\") pod \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\" (UID: \"5a4e0482-cbbb-4eb9-973d-2774368f8c89\") "
Nov 02 17:50:13.582870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.569522    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9b6e90-ee95-4f05-85db-6fef1576215b-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:13.582870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.569561    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a4e0482-cbbb-4eb9-973d-2774368f8c89-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:13.622866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.622024    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp" pod "9c9b6e90-ee95-4f05-85db-6fef1576215b" (UID: "9c9b6e90-ee95-4f05-85db-6fef1576215b"). InnerVolumeSpecName "kube-api-access-z66tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:13.624512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.623826    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn" pod "5a4e0482-cbbb-4eb9-973d-2774368f8c89" (UID: "5a4e0482-cbbb-4eb9-973d-2774368f8c89"). InnerVolumeSpecName "kube-api-access-4t8cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:13.631217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.631176    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93435d3-2314-4f81-ac77-5edfd7cefa67" path="/var/lib/kubelet/pods/c93435d3-2314-4f81-ac77-5edfd7cefa67/volumes"
Nov 02 17:50:13.671893 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.671829    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66tp\" (UniqueName: \"kubernetes.io/projected/9c9b6e90-ee95-4f05-85db-6fef1576215b-kube-api-access-z66tp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:13.672061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:13.671899    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8cn\" (UniqueName: \"kubernetes.io/projected/5a4e0482-cbbb-4eb9-973d-2774368f8c89-kube-api-access-4t8cn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.176939    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"]
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.177947    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9c9b6e90-ee95-4f05-85db-6fef1576215b" containerName="payload-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.177965    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9b6e90-ee95-4f05-85db-6fef1576215b" containerName="payload-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.177995    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5a4e0482-cbbb-4eb9-973d-2774368f8c89" containerName="payload-pre-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.178003    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4e0482-cbbb-4eb9-973d-2774368f8c89" containerName="payload-pre-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.178025    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c93435d3-2314-4f81-ac77-5edfd7cefa67" containerName="payload-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.178033    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93435d3-2314-4f81-ac77-5edfd7cefa67" containerName="payload-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.178182    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9b6e90-ee95-4f05-85db-6fef1576215b" containerName="payload-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.178201    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4e0482-cbbb-4eb9-973d-2774368f8c89" containerName="payload-pre-processing"
Nov 02 17:50:14.180885 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.178214    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93435d3-2314-4f81-ac77-5edfd7cefa67" containerName="payload-processing"
Nov 02 17:50:14.186776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.185792    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.186776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.186546    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"]
Nov 02 17:50:14.192797 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.192768    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8f7957\""
Nov 02 17:50:14.193000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.192958    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8f7957-dockercfg-xbk65\""
Nov 02 17:50:14.243962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.243934    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"
Nov 02 17:50:14.244460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.244428    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc" event={"ID":"5a4e0482-cbbb-4eb9-973d-2774368f8c89","Type":"ContainerDied","Data":"5e3b246854ccb38313111e43baeea57c6e7a56c84d177b37dd848589ea9b3e76"}
Nov 02 17:50:14.244585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.244502    5042 scope.go:122] "RemoveContainer" containerID="081a67310e01db93573cc86abebc35635ad0ab35bb9e25615267642cf60e64f5"
Nov 02 17:50:14.275826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.275793    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"
Nov 02 17:50:14.277483 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.274184    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz" event={"ID":"9c9b6e90-ee95-4f05-85db-6fef1576215b","Type":"ContainerDied","Data":"8339dfa99075b1459efd48728804ea566e12c7d093d0d486460077eb368ef3c2"}
Nov 02 17:50:14.280008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.279983    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"]
Nov 02 17:50:14.288142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.288085    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dd1cba-847946cbdc-gqxkc"]
Nov 02 17:50:14.289870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.288521    5042 scope.go:122] "RemoveContainer" containerID="c8d5fe16988ad84ce3cf955994c231ec2c7ca69556729bf50168cac6a2e2996b"
Nov 02 17:50:14.308066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.308039    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"]
Nov 02 17:50:14.316171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.316132    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dd1cba-7c57d9bd65-t4wpz"]
Nov 02 17:50:14.333619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.333574    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clx65\" (UniqueName: \"kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65\") pod \"payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.333910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.333887    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.435734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.435694    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.435915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.435828    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clx65\" (UniqueName: \"kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65\") pod \"payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.436547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.436524    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.445646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.445619    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clx65\" (UniqueName: \"kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65\") pod \"payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.545706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.545664    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:14.860250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:14.860219    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"]
Nov 02 17:50:14.860643 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:50:14.860622    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0869f30c_b58a_40ee_9108_d4c504934fd7.slice/crio-f69a070100ace9ebb49275844008e6c735abec65336e8c68030e2cf5472a1fc0 WatchSource:0}: Error finding container f69a070100ace9ebb49275844008e6c735abec65336e8c68030e2cf5472a1fc0: Status 404 returned error can't find the container with id f69a070100ace9ebb49275844008e6c735abec65336e8c68030e2cf5472a1fc0
Nov 02 17:50:15.285133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:15.285094    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj" event={"ID":"0869f30c-b58a-40ee-9108-d4c504934fd7","Type":"ContainerStarted","Data":"bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16"}
Nov 02 17:50:15.285645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:15.285160    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:15.285645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:15.285181    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj" event={"ID":"0869f30c-b58a-40ee-9108-d4c504934fd7","Type":"ContainerStarted","Data":"f69a070100ace9ebb49275844008e6c735abec65336e8c68030e2cf5472a1fc0"}
Nov 02 17:50:15.307191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:15.307140    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj" podStartSLOduration=1.307126536 podStartE2EDuration="1.307126536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:15.30258133 +0000 UTC m=+1027.881990717" watchObservedRunningTime="2026-11-02 17:50:15.307126536 +0000 UTC m=+1027.886535923"
Nov 02 17:50:15.545981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:15.545890    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4e0482-cbbb-4eb9-973d-2774368f8c89" path="/var/lib/kubelet/pods/5a4e0482-cbbb-4eb9-973d-2774368f8c89/volumes"
Nov 02 17:50:15.546446 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:15.546429    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9b6e90-ee95-4f05-85db-6fef1576215b" path="/var/lib/kubelet/pods/9c9b6e90-ee95-4f05-85db-6fef1576215b/volumes"
Nov 02 17:50:18.548221 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:50:18.548136    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:50:21.036734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:21.036699    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"]
Nov 02 17:50:21.041598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:21.041572    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7d08855-hjb4c"]
Nov 02 17:50:21.544614 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:21.544584    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4465b7-fd5c-4511-8a30-57f3f52ffc1d" path="/var/lib/kubelet/pods/da4465b7-fd5c-4511-8a30-57f3f52ffc1d/volumes"
Nov 02 17:50:22.032365 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:22.032328    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"]
Nov 02 17:50:22.037437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:22.037402    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a983067e-xpvf2"]
Nov 02 17:50:23.549003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:23.548966    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd425fb7-693f-43ea-836d-a5575d4b17c2" path="/var/lib/kubelet/pods/dd425fb7-693f-43ea-836d-a5575d4b17c2/volumes"
Nov 02 17:50:26.428036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:26.428004    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:50:29.520873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:29.520807    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"]
Nov 02 17:50:29.522303 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:29.522264    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" containerID="cri-o://9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae" gracePeriod=30
Nov 02 17:50:31.172872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:31.160742    5042 patch_prober.go:29] interesting pod/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:50:31.172872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:31.160800    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:50:34.228300 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.228260    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"]
Nov 02 17:50:34.255177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.255136    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"]
Nov 02 17:50:34.255383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.255341    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.300980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.300932    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsdsc\" (UniqueName: \"kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.301175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.301033    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.358149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.358115    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c8832b\""
Nov 02 17:50:34.408216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.408164    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsdsc\" (UniqueName: \"kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.408608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.408562    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.410162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.410142    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.451067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.450758    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsdsc\" (UniqueName: \"kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc\") pod \"payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.608211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.608170    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:34.952593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.952560    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"]
Nov 02 17:50:34.952751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:34.952619    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:50:34.959269 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:50:34.959243    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8811ce5_839d_4fae_bd3f_f3f3118b5667.slice/crio-4d257fbef14411ec3c00bec6ac117e1471a49423885016a18b79445cb563098b WatchSource:0}: Error finding container 4d257fbef14411ec3c00bec6ac117e1471a49423885016a18b79445cb563098b: Status 404 returned error can't find the container with id 4d257fbef14411ec3c00bec6ac117e1471a49423885016a18b79445cb563098b
Nov 02 17:50:35.026362 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.026323    5042 generic.go:356] "Generic (PLEG): container finished" podID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerID="9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae" exitCode=0
Nov 02 17:50:35.026524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.026498    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" event={"ID":"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1","Type":"ContainerDied","Data":"9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae"}
Nov 02 17:50:35.026567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.026540    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c" event={"ID":"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1","Type":"ContainerDied","Data":"a4dadc692f76742faca8e7645d15286f76f7b21da3a74d68e2f64f72fb0b2127"}
Nov 02 17:50:35.026567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.026560    5042 scope.go:122] "RemoveContainer" containerID="9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae"
Nov 02 17:50:35.026630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.026602    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031198    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4" event={"ID":"f8811ce5-839d-4fae-bd3f-f3f3118b5667","Type":"ContainerStarted","Data":"4d257fbef14411ec3c00bec6ac117e1471a49423885016a18b79445cb563098b"}
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031364    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031423    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031480    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031562    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031587    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031667    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031711    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031799    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.032246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.031865    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket\") pod \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\" (UID: \"4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1\") "
Nov 02 17:50:35.036270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.036233    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:35.036681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.036660    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:35.038399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.037170    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:35.039571 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.039526    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:35.041366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.041295    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:35.047201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.047140    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"]
Nov 02 17:50:35.054130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.054074    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09e6875-tztm6"]
Nov 02 17:50:35.076854 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.076777    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:35.078793 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.078755    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "kube-api-access-9r4nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:35.087192 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.087161    5042 scope.go:122] "RemoveContainer" containerID="9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae"
Nov 02 17:50:35.087771 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:50:35.087694    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae\": container with ID starting with 9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae not found: ID does not exist" containerID="9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae"
Nov 02 17:50:35.087771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.087738    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae"} err="failed to get container status \"9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae\": rpc error: code = NotFound desc = could not find container \"9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae\": container with ID starting with 9d34120e5e413ef257d988c13d427749981db7decc607778f56c982b6c6c8aae not found: ID does not exist"
Nov 02 17:50:35.094599 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.092119    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:35.109126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.095017    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy" pod "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" (UID: "4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139338    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139381    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139396    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139410    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139424    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139437    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139451    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139465    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r4nf\" (UniqueName: \"kubernetes.io/projected/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-kube-api-access-9r4nf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.140413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.139479    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:35.370570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.370529    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"]
Nov 02 17:50:35.375527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.375494    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-da5f07e6-openshift-default-577cff9d8f-q8r8c"]
Nov 02 17:50:35.544661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.544576    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" path="/var/lib/kubelet/pods/4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1/volumes"
Nov 02 17:50:35.545139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.545123    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955d6a99-dd07-47e9-af0d-4afd4f515be5" path="/var/lib/kubelet/pods/955d6a99-dd07-47e9-af0d-4afd4f515be5/volumes"
Nov 02 17:50:35.764679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.764642    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"]
Nov 02 17:50:35.765371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.765354    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy"
Nov 02 17:50:35.765437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.765373    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy"
Nov 02 17:50:35.765519 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.765509    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa5d96e-c2a9-4df7-8d00-dd4b39a7caf1" containerName="istio-proxy"
Nov 02 17:50:35.769508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.769490    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:35.781645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.781612    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"]
Nov 02 17:50:35.782287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.782262    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cd2738-dockercfg-xx5h8\""
Nov 02 17:50:35.782419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.782347    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cd2738\""
Nov 02 17:50:35.808977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.808889    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"]
Nov 02 17:50:35.828061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.828030    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"]
Nov 02 17:50:35.828279 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.828142    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:35.850828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.850791    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqb8\" (UniqueName: \"kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8\") pod \"payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:35.851058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.850872    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:35.954877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.954820    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:35.955253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.955229    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvdm\" (UniqueName: \"kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm\") pod \"payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:35.955640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.955620    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqb8\" (UniqueName: \"kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8\") pod \"payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:35.957461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.957443    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:35.966274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:35.965286    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:36.020863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.018987    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqb8\" (UniqueName: \"kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8\") pod \"payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:36.075919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.070020    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:36.075919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.070078    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvdm\" (UniqueName: \"kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm\") pod \"payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:36.075919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.070880    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:36.092859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.091753    5042 generic.go:356] "Generic (PLEG): container finished" podID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerID="9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71" exitCode=0
Nov 02 17:50:36.092859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.091908    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" event={"ID":"c15640b9-72f5-4254-a525-e72f7a75a24e","Type":"ContainerDied","Data":"9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"}
Nov 02 17:50:36.092859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.092482    5042 scope.go:122] "RemoveContainer" containerID="9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"
Nov 02 17:50:36.105356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.097244    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvdm\" (UniqueName: \"kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm\") pod \"payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:36.105356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.098020    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.105356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.098106    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.105356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.098149    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.105356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.098661    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:36.112936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.111829    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4" event={"ID":"f8811ce5-839d-4fae-bd3f-f3f3118b5667","Type":"ContainerStarted","Data":"47a37d7007549ec1ae94a884bc71425006c9a56952c3caefc82088df64d7f401"}
Nov 02 17:50:36.116946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.113262    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:36.148578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.148544    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:36.196181 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.196121    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4" podStartSLOduration=2.196102576 podStartE2EDuration="2.196102576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:36.188053526 +0000 UTC m=+1048.767462942" watchObservedRunningTime="2026-11-02 17:50:36.196102576 +0000 UTC m=+1048.775511964"
Nov 02 17:50:36.625273 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:50:36.625238    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62215556_fff0_4c26_a7d3_681dbe5a6d1c.slice/crio-e59d92b6f9df58e2a20186a11a98f026d435723624768ee99f0ffb0d74652b65 WatchSource:0}: Error finding container e59d92b6f9df58e2a20186a11a98f026d435723624768ee99f0ffb0d74652b65: Status 404 returned error can't find the container with id e59d92b6f9df58e2a20186a11a98f026d435723624768ee99f0ffb0d74652b65
Nov 02 17:50:36.628453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.628418    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"]
Nov 02 17:50:36.759705 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:50:36.759663    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04657e80_f3ed_443a_986f_7ae228f4a891.slice/crio-c76d0ecbe0c37f7f984ee48244f1070f2b7c98414f563d86d936d7b5ebd10369 WatchSource:0}: Error finding container c76d0ecbe0c37f7f984ee48244f1070f2b7c98414f563d86d936d7b5ebd10369: Status 404 returned error can't find the container with id c76d0ecbe0c37f7f984ee48244f1070f2b7c98414f563d86d936d7b5ebd10369
Nov 02 17:50:36.761959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:36.761768    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"]
Nov 02 17:50:37.216084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.214357    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc" event={"ID":"62215556-fff0-4c26-a7d3-681dbe5a6d1c","Type":"ContainerStarted","Data":"811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157"}
Nov 02 17:50:37.216084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.214411    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc" event={"ID":"62215556-fff0-4c26-a7d3-681dbe5a6d1c","Type":"ContainerStarted","Data":"e59d92b6f9df58e2a20186a11a98f026d435723624768ee99f0ffb0d74652b65"}
Nov 02 17:50:37.216084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.214954    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:37.217159 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.217128    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg" event={"ID":"04657e80-f3ed-443a-986f-7ae228f4a891","Type":"ContainerStarted","Data":"14c128ad0479b94c27a52751b1f61d6a33f61cf46a2d76e80cc4cdf8148abfee"}
Nov 02 17:50:37.217238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.217207    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:37.217238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.217233    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg" event={"ID":"04657e80-f3ed-443a-986f-7ae228f4a891","Type":"ContainerStarted","Data":"c76d0ecbe0c37f7f984ee48244f1070f2b7c98414f563d86d936d7b5ebd10369"}
Nov 02 17:50:37.219914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.219892    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" event={"ID":"c15640b9-72f5-4254-a525-e72f7a75a24e","Type":"ContainerStarted","Data":"2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60"}
Nov 02 17:50:37.234876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.234214    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc" podStartSLOduration=2.234195153 podStartE2EDuration="2.234195153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:37.233244609 +0000 UTC m=+1049.812653998" watchObservedRunningTime="2026-11-02 17:50:37.234195153 +0000 UTC m=+1049.813604541"
Nov 02 17:50:37.293863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:37.284266    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg" podStartSLOduration=2.284245967 podStartE2EDuration="2.284245967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:37.283236227 +0000 UTC m=+1049.862645613" watchObservedRunningTime="2026-11-02 17:50:37.284245967 +0000 UTC m=+1049.863655355"
Nov 02 17:50:38.031614 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:38.031578    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"]
Nov 02 17:50:38.035888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:38.035860    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f82bf2f-zmrdl"]
Nov 02 17:50:38.057164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:38.057126    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:50:38.058311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:38.058286    5042 patch_prober.go:29] interesting pod/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 17:50:38.058415 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:38.058337    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 17:50:39.058047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:39.058011    5042 patch_prober.go:29] interesting pod/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 17:50:39.058450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:39.058067    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 17:50:39.545827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:39.545792    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c3ea23-26d7-418c-8375-e7e9f959b7c3" path="/var/lib/kubelet/pods/65c3ea23-26d7-418c-8375-e7e9f959b7c3/volumes"
Nov 02 17:50:40.057392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:40.057356    5042 patch_prober.go:29] interesting pod/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 17:50:40.057697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:40.057421    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 17:50:41.058164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:41.058128    5042 patch_prober.go:29] interesting pod/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 17:50:41.058554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:41.058191    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 17:50:42.057679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:42.057635    5042 patch_prober.go:29] interesting pod/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Nov 02 17:50:42.057907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:42.057705    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Nov 02 17:50:42.993774 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:42.993739    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:50:43.070755 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:43.070721    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:50:43.321620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:43.321533    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:50:48.232016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:48.231976    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:50:48.232685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:48.232664    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:50:48.233095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:48.233074    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:50:48.542613 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:50:48.542524    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:50:50.350526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:50.349029    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"]
Nov 02 17:50:50.351560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:50.351486    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv" podUID="e586ed01-04be-43cf-bb26-78f6bf030ea8" containerName="payload-processing" containerID="cri-o://c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6" gracePeriod=30
Nov 02 17:50:50.373751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:50.373713    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"]
Nov 02 17:50:50.374038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:50.374008    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb" podUID="93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" containerName="payload-pre-processing" containerID="cri-o://608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087" gracePeriod=30
Nov 02 17:50:51.041363 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.041287    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:50:51.050924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.050829    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"]
Nov 02 17:50:51.058713 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.058657    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-849cf249-ssnnq"]
Nov 02 17:50:51.082491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.082460    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5\" (UniqueName: \"kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5\") pod \"e586ed01-04be-43cf-bb26-78f6bf030ea8\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") "
Nov 02 17:50:51.082813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.082794    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume\") pod \"e586ed01-04be-43cf-bb26-78f6bf030ea8\" (UID: \"e586ed01-04be-43cf-bb26-78f6bf030ea8\") "
Nov 02 17:50:51.083604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.083577    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume" pod "e586ed01-04be-43cf-bb26-78f6bf030ea8" (UID: "e586ed01-04be-43cf-bb26-78f6bf030ea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:51.086306 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.086282    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5" pod "e586ed01-04be-43cf-bb26-78f6bf030ea8" (UID: "e586ed01-04be-43cf-bb26-78f6bf030ea8"). InnerVolumeSpecName "kube-api-access-5l2k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:51.128158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.128129    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:50:51.184566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.184471    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume\") pod \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") "
Nov 02 17:50:51.184830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.184806    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln\" (UniqueName: \"kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln\") pod \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\" (UID: \"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506\") "
Nov 02 17:50:51.185625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.185590    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e586ed01-04be-43cf-bb26-78f6bf030ea8-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:51.185625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.185620    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l2k5\" (UniqueName: \"kubernetes.io/projected/e586ed01-04be-43cf-bb26-78f6bf030ea8-kube-api-access-5l2k5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:51.186361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.186331    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume" pod "93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" (UID: "93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:51.189235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.189202    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln" pod "93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" (UID: "93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506"). InnerVolumeSpecName "kube-api-access-clfln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:51.287335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.287299    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfln\" (UniqueName: \"kubernetes.io/projected/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-kube-api-access-clfln\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:51.287335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.287328    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:51.420498 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.420452    5042 generic.go:356] "Generic (PLEG): container finished" podID="93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" containerID="608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087" exitCode=0
Nov 02 17:50:51.420996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.420514    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"
Nov 02 17:50:51.420996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.420518    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb" event={"ID":"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506","Type":"ContainerDied","Data":"608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087"}
Nov 02 17:50:51.420996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.420582    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb" event={"ID":"93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506","Type":"ContainerDied","Data":"812a70c7ce82ef88235b637786eb40c6bacf6caa343567beb67efcb7c48ca8a8"}
Nov 02 17:50:51.420996 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.420617    5042 scope.go:122] "RemoveContainer" containerID="608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087"
Nov 02 17:50:51.422433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.422408    5042 generic.go:356] "Generic (PLEG): container finished" podID="e586ed01-04be-43cf-bb26-78f6bf030ea8" containerID="c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6" exitCode=0
Nov 02 17:50:51.422550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.422456    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"
Nov 02 17:50:51.422550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.422498    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv" event={"ID":"e586ed01-04be-43cf-bb26-78f6bf030ea8","Type":"ContainerDied","Data":"c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6"}
Nov 02 17:50:51.422647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.422552    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv" event={"ID":"e586ed01-04be-43cf-bb26-78f6bf030ea8","Type":"ContainerDied","Data":"4524a0c859ea9315321b831ef8961d23b695c4dedc9d734c151a56b8274ed2a9"}
Nov 02 17:50:51.433925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.433905    5042 scope.go:122] "RemoveContainer" containerID="608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087"
Nov 02 17:50:51.434243 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:50:51.434223    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087\": container with ID starting with 608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087 not found: ID does not exist" containerID="608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087"
Nov 02 17:50:51.434318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.434258    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087"} err="failed to get container status \"608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087\": rpc error: code = NotFound desc = could not find container \"608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087\": container with ID starting with 608e209f263677905496d60acec4dd36e40cfc35e26bad3002e0cc401fcf1087 not found: ID does not exist"
Nov 02 17:50:51.434318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.434279    5042 scope.go:122] "RemoveContainer" containerID="c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6"
Nov 02 17:50:51.447056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.447029    5042 scope.go:122] "RemoveContainer" containerID="c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6"
Nov 02 17:50:51.447417 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:50:51.447392    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6\": container with ID starting with c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6 not found: ID does not exist" containerID="c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6"
Nov 02 17:50:51.447505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.447432    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6"} err="failed to get container status \"c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6\": rpc error: code = NotFound desc = could not find container \"c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6\": container with ID starting with c7f7735404b32c7711d5dbc671e83ea98efb5dba8d3d2bc333da0f6e7b6285b6 not found: ID does not exist"
Nov 02 17:50:51.455002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.454976    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"]
Nov 02 17:50:51.463114 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.463082    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d1fc77-7dc6cdbdc-wzztb"]
Nov 02 17:50:51.494343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.494301    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"]
Nov 02 17:50:51.494549 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.494370    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d1fc77-96dbb44c8-sfzqv"]
Nov 02 17:50:51.546091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.546059    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2805413d-aa54-45dc-826d-c68d83d19eba" path="/var/lib/kubelet/pods/2805413d-aa54-45dc-826d-c68d83d19eba/volumes"
Nov 02 17:50:51.546687 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.546667    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" path="/var/lib/kubelet/pods/93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506/volumes"
Nov 02 17:50:51.547074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:51.547059    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e586ed01-04be-43cf-bb26-78f6bf030ea8" path="/var/lib/kubelet/pods/e586ed01-04be-43cf-bb26-78f6bf030ea8/volumes"
Nov 02 17:50:53.033022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:53.032945    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"]
Nov 02 17:50:53.037949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:53.037916    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a112fd1-f2r28"]
Nov 02 17:50:53.545972 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:53.545942    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6db4524-c25c-4d26-9e3a-360cf0d72632" path="/var/lib/kubelet/pods/e6db4524-c25c-4d26-9e3a-360cf0d72632/volumes"
Nov 02 17:50:58.897694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:58.897584    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"]
Nov 02 17:50:58.898343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:58.898263    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f" podUID="7c7d3815-1deb-46a6-92be-dfed9a7139a0" containerName="payload-pre-processing" containerID="cri-o://b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5" gracePeriod=30
Nov 02 17:50:59.473015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.472978    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:50:59.548863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.548411    5042 generic.go:356] "Generic (PLEG): container finished" podID="7c7d3815-1deb-46a6-92be-dfed9a7139a0" containerID="b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5" exitCode=0
Nov 02 17:50:59.548863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.548554    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"
Nov 02 17:50:59.557864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.555774    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f" event={"ID":"7c7d3815-1deb-46a6-92be-dfed9a7139a0","Type":"ContainerDied","Data":"b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5"}
Nov 02 17:50:59.557864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.555834    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f" event={"ID":"7c7d3815-1deb-46a6-92be-dfed9a7139a0","Type":"ContainerDied","Data":"5ddbcd5f8d8bedce0b2a6f90e3ce0f027d275aac99ee93bd9604d9d45a26cfe0"}
Nov 02 17:50:59.557864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.555907    5042 scope.go:122] "RemoveContainer" containerID="b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5"
Nov 02 17:50:59.576877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.576025    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr\" (UniqueName: \"kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr\") pod \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") "
Nov 02 17:50:59.576877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.576153    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume\") pod \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\" (UID: \"7c7d3815-1deb-46a6-92be-dfed9a7139a0\") "
Nov 02 17:50:59.578177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.577270    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume" pod "7c7d3815-1deb-46a6-92be-dfed9a7139a0" (UID: "7c7d3815-1deb-46a6-92be-dfed9a7139a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:59.597872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.594712    5042 scope.go:122] "RemoveContainer" containerID="b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5"
Nov 02 17:50:59.597872 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:50:59.597761    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5\": container with ID starting with b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5 not found: ID does not exist" containerID="b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5"
Nov 02 17:50:59.597872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.597802    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5"} err="failed to get container status \"b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5\": rpc error: code = NotFound desc = could not find container \"b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5\": container with ID starting with b197dc30d0c1c0aec6a8adfce5e3373590e576a44be3e34e538ad70d5f57c6b5 not found: ID does not exist"
Nov 02 17:50:59.598456 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.598412    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr" pod "7c7d3815-1deb-46a6-92be-dfed9a7139a0" (UID: "7c7d3815-1deb-46a6-92be-dfed9a7139a0"). InnerVolumeSpecName "kube-api-access-b9vqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:59.678028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.677945    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vqr\" (UniqueName: \"kubernetes.io/projected/7c7d3815-1deb-46a6-92be-dfed9a7139a0-kube-api-access-b9vqr\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:59.678028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.677975    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7d3815-1deb-46a6-92be-dfed9a7139a0-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:50:59.885635 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.885595    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"]
Nov 02 17:50:59.893855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:50:59.893689    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-82b305-7c58b8cf9f-ngg2f"]
Nov 02 17:51:00.869077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.869039    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"]
Nov 02 17:51:00.870000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.869982    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7c7d3815-1deb-46a6-92be-dfed9a7139a0" containerName="payload-pre-processing"
Nov 02 17:51:00.870048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870003    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7d3815-1deb-46a6-92be-dfed9a7139a0" containerName="payload-pre-processing"
Nov 02 17:51:00.870048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870033    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e586ed01-04be-43cf-bb26-78f6bf030ea8" containerName="payload-processing"
Nov 02 17:51:00.870048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870041    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e586ed01-04be-43cf-bb26-78f6bf030ea8" containerName="payload-processing"
Nov 02 17:51:00.870158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870061    5042 cpu_manager.go:414] "containerMap: removing container" podUID="93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" containerName="payload-pre-processing"
Nov 02 17:51:00.870158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870069    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" containerName="payload-pre-processing"
Nov 02 17:51:00.870259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870249    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7d3815-1deb-46a6-92be-dfed9a7139a0" containerName="payload-pre-processing"
Nov 02 17:51:00.870294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870267    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e586ed01-04be-43cf-bb26-78f6bf030ea8" containerName="payload-processing"
Nov 02 17:51:00.870294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.870279    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c6229a-5c06-4fd4-a8c5-e6ee7d3cd506" containerName="payload-pre-processing"
Nov 02 17:51:00.876013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.875988    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.889779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.886822    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"]
Nov 02 17:51:00.903935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.903727    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.903935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.903785    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.903935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.903883    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwqn9\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.904312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.903948    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.904312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.904002    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.904312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.904098    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.904312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.904126    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.904312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.904152    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.904312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.904176    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:00.923608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:00.923565    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-3568c5-openshift-default-dockercfg-w8fhr\""
Nov 02 17:51:01.039527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.037923    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.054868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.040388    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.055168 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.041197    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.061761    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.061860    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.061890    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.062109    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.062145    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.062240    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwqn9\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.062314    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.062383    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.070065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.069599    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.086926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.073357    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.086926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.073615    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.086926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.074635    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.165982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.158812    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwqn9\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.165982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.161790    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.192911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.190717    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy\") pod \"e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") " pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.226435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.226393    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:01.276995 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.272102    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"]
Nov 02 17:51:01.300647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.300584    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3da22a0b-rs7qr"]
Nov 02 17:51:01.548705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.548668    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e501d6-d895-471e-aa91-a4c3bcad5e76" path="/var/lib/kubelet/pods/54e501d6-d895-471e-aa91-a4c3bcad5e76/volumes"
Nov 02 17:51:01.549287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.549260    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7d3815-1deb-46a6-92be-dfed9a7139a0" path="/var/lib/kubelet/pods/7c7d3815-1deb-46a6-92be-dfed9a7139a0/volumes"
Nov 02 17:51:01.590398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.590359    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"]
Nov 02 17:51:01.609058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.609010    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:01.609186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.609111    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:01.609186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:01.609156    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:02.583946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:02.583902    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" event={"ID":"14ba8a64-73a1-4122-b4f0-d3b1579aaad4","Type":"ContainerStarted","Data":"ca5773e640611b6702517d50f03098feeeeae145efc737b4cd5620a0488d2063"}
Nov 02 17:51:02.584632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:02.584604    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" event={"ID":"14ba8a64-73a1-4122-b4f0-d3b1579aaad4","Type":"ContainerStarted","Data":"f899c643fcfe99d57e48362d8ae4fd2c9830867574f8d8d29f0f5b088c17792d"}
Nov 02 17:51:02.608586 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:02.608530    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podStartSLOduration=2.608513655 podStartE2EDuration="2.608513655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:02.605386042 +0000 UTC m=+1075.184795441" watchObservedRunningTime="2026-11-02 17:51:02.608513655 +0000 UTC m=+1075.187923043"
Nov 02 17:51:03.227111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:03.227075    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:03.233174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:03.233147    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:03.614918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:03.614300    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:03.615634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:03.615603    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:04.047910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:04.046278    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"]
Nov 02 17:51:04.051864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:04.049188    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6c47015-4szjx"]
Nov 02 17:51:05.546288 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:05.546252    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0adafecb-5cdd-4a57-9230-80a649be16a3" path="/var/lib/kubelet/pods/0adafecb-5cdd-4a57-9230-80a649be16a3/volumes"
Nov 02 17:51:08.031743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:08.031707    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"]
Nov 02 17:51:08.036136 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:08.036100    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab722c4e-cwnnt"]
Nov 02 17:51:08.404810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:08.404714    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"]
Nov 02 17:51:08.405371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:08.405337    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy" containerID="cri-o://78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61" gracePeriod=30
Nov 02 17:51:09.261986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:09.261949    5042 scope.go:122] "RemoveContainer" containerID="de7fe871569b59fd8b61d672eebb650f6e65d93e4282c991baae1dd1805fa67f"
Nov 02 17:51:09.273506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:09.273479    5042 scope.go:122] "RemoveContainer" containerID="9c6d6ed5575a89587b80dedceca80d1f4c34eba6d2d3adf96526bc87baabdaef"
Nov 02 17:51:09.283777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:09.283752    5042 scope.go:122] "RemoveContainer" containerID="e51f50c2fd9e9002480059193ab20017dc6ac1f2d7d091d44b02857bc0de1148"
Nov 02 17:51:09.298075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:09.298053    5042 scope.go:122] "RemoveContainer" containerID="808314e5919f071f4f38b24741a39d9eb320425c364f9c8f9bf7e62578a62322"
Nov 02 17:51:09.545461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:09.545359    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca8fa5d-f60c-4367-8d20-7c8967d941b0" path="/var/lib/kubelet/pods/6ca8fa5d-f60c-4367-8d20-7c8967d941b0/volumes"
Nov 02 17:51:11.040692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:11.040653    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"]
Nov 02 17:51:11.045325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:11.045291    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0ed3068-nvdt7"]
Nov 02 17:51:11.546527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:11.546488    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4" path="/var/lib/kubelet/pods/7f5eba0f-3008-4e68-a5b0-d4e96ec4b0d4/volumes"
Nov 02 17:51:13.724272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.724245    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:51:13.830717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.830687    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.830926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.830784    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.830926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.830867    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.830926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.830895    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.831081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.830928    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.831081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.830981    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.831081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831035    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.831231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831081    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:13.831231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831100    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.831343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc\") pod \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\" (UID: \"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33\") "
Nov 02 17:51:13.831343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831326    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:13.831458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831381    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:13.831458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831416    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:13.831872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.831813    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:13.832316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.832286    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.832316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.832315    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.832489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.832330    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.832489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.832345    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.832489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.832358    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.834107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.834076    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:13.834591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.834568    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:51:13.834816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.834794    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:13.836261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.836235    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc" pod "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" (UID: "cecca6b1-7fc9-44cc-9bf8-ba70ee467f33"). InnerVolumeSpecName "kube-api-access-m6gxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:13.839830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.839799    5042 generic.go:356] "Generic (PLEG): container finished" podID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerID="78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61" exitCode=0
Nov 02 17:51:13.840024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.840004    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" event={"ID":"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33","Type":"ContainerDied","Data":"78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61"}
Nov 02 17:51:13.840095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.840018    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"
Nov 02 17:51:13.840095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.840061    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth" event={"ID":"cecca6b1-7fc9-44cc-9bf8-ba70ee467f33","Type":"ContainerDied","Data":"25e6485d20f8389e983fa29e0c1a0b2644d270f6f50c6a0b050d145ef5f1e8f0"}
Nov 02 17:51:13.840095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.840086    5042 scope.go:122] "RemoveContainer" containerID="78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61"
Nov 02 17:51:13.869758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.869734    5042 scope.go:122] "RemoveContainer" containerID="0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.883458    5042 scope.go:122] "RemoveContainer" containerID="78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61"
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:13.883885    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61\": container with ID starting with 78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61 not found: ID does not exist" containerID="78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61"
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.883909    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"]
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.883922    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61"} err="failed to get container status \"78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61\": rpc error: code = NotFound desc = could not find container \"78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61\": container with ID starting with 78a25fc18dbcfb096e3957393ee40d37bc29dce19f58a2823128b93b095c5d61 not found: ID does not exist"
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.883950    5042 scope.go:122] "RemoveContainer" containerID="0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:13.884331    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8\": container with ID starting with 0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8 not found: ID does not exist" containerID="0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"
Nov 02 17:51:13.885188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.884353    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8"} err="failed to get container status \"0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8\": rpc error: code = NotFound desc = could not find container \"0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8\": container with ID starting with 0b7019e7ce0d9635ba9d0fe50068a21bdbdfc6321a07b0fe4abc8d9c685e73d8 not found: ID does not exist"
Nov 02 17:51:13.888473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.888444    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-82b305-openshift-default-7bb775b8c5-b6rth"]
Nov 02 17:51:13.933675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.933629    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.933675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.933665    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.933675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.933680    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gxc\" (UniqueName: \"kubernetes.io/projected/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-kube-api-access-m6gxc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:13.933937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:13.933694    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:15.646158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:15.646124    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" path="/var/lib/kubelet/pods/cecca6b1-7fc9-44cc-9bf8-ba70ee467f33/volumes"
Nov 02 17:51:16.641616 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:16.641453    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"]
Nov 02 17:51:16.641820 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:16.641657    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4" podUID="f8811ce5-839d-4fae-bd3f-f3f3118b5667" containerName="payload-pre-processing" containerID="cri-o://47a37d7007549ec1ae94a884bc71425006c9a56952c3caefc82088df64d7f401" gracePeriod=30
Nov 02 17:51:17.089717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.089677    5042 generic.go:356] "Generic (PLEG): container finished" podID="f8811ce5-839d-4fae-bd3f-f3f3118b5667" containerID="47a37d7007549ec1ae94a884bc71425006c9a56952c3caefc82088df64d7f401" exitCode=0
Nov 02 17:51:17.090680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.090653    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4" event={"ID":"f8811ce5-839d-4fae-bd3f-f3f3118b5667","Type":"ContainerDied","Data":"47a37d7007549ec1ae94a884bc71425006c9a56952c3caefc82088df64d7f401"}
Nov 02 17:51:17.207982 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.207812    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:51:17.343730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.343644    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc\" (UniqueName: \"kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc\") pod \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") "
Nov 02 17:51:17.343935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.343825    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume\") pod \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\" (UID: \"f8811ce5-839d-4fae-bd3f-f3f3118b5667\") "
Nov 02 17:51:17.344160 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.344128    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume" pod "f8811ce5-839d-4fae-bd3f-f3f3118b5667" (UID: "f8811ce5-839d-4fae-bd3f-f3f3118b5667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:17.344315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.344301    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8811ce5-839d-4fae-bd3f-f3f3118b5667-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:17.346050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.346028    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc" pod "f8811ce5-839d-4fae-bd3f-f3f3118b5667" (UID: "f8811ce5-839d-4fae-bd3f-f3f3118b5667"). InnerVolumeSpecName "kube-api-access-fsdsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:17.446754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:17.446716    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsdsc\" (UniqueName: \"kubernetes.io/projected/f8811ce5-839d-4fae-bd3f-f3f3118b5667-kube-api-access-fsdsc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:18.104406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:18.104375    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"
Nov 02 17:51:18.105000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:18.104402    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4" event={"ID":"f8811ce5-839d-4fae-bd3f-f3f3118b5667","Type":"ContainerDied","Data":"4d257fbef14411ec3c00bec6ac117e1471a49423885016a18b79445cb563098b"}
Nov 02 17:51:18.105000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:18.104479    5042 scope.go:122] "RemoveContainer" containerID="47a37d7007549ec1ae94a884bc71425006c9a56952c3caefc82088df64d7f401"
Nov 02 17:51:18.129076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:18.129042    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"]
Nov 02 17:51:18.134720 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:18.134689    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c8832b-64fbbb4fd7-htzj4"]
Nov 02 17:51:18.564236 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:18.564194    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:51:19.545149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:19.545115    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8811ce5-839d-4fae-bd3f-f3f3118b5667" path="/var/lib/kubelet/pods/f8811ce5-839d-4fae-bd3f-f3f3118b5667/volumes"
Nov 02 17:51:20.827371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.827322    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"]
Nov 02 17:51:20.828964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.828932    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy"
Nov 02 17:51:20.828964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.828963    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829036    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f8811ce5-839d-4fae-bd3f-f3f3118b5667" containerName="payload-pre-processing"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829045    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8811ce5-839d-4fae-bd3f-f3f3118b5667" containerName="payload-pre-processing"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829057    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829066    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829246    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829268    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8811ce5-839d-4fae-bd3f-f3f3118b5667" containerName="payload-pre-processing"
Nov 02 17:51:20.830970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.829278    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cecca6b1-7fc9-44cc-9bf8-ba70ee467f33" containerName="istio-proxy"
Nov 02 17:51:20.854584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.854543    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"]
Nov 02 17:51:20.854773 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.854669    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.865349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.865323    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-e4fdc1-openshift-default-dockercfg-klg2x\""
Nov 02 17:51:20.920622 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.920580    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.920818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.920691    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.920818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.920729    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.920988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.920958    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.921059 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.921042    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.921126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.921081    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.921126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.921117    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.921281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.921257    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:20.921435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:20.921409    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwhs\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.022919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.022860    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwhs\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.022982    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023064    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023106    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023247    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023304    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023338    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023376    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.023437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.023423    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.024440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.024410    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.032970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.025054    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.032970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.025284    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.032970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.025515    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.032970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.025744    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.053143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.052121    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.065863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.065793    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.066047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.065891    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"]
Nov 02 17:51:21.073861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.066986    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwhs\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.098164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.098056    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984f0fdd-6tkg5"]
Nov 02 17:51:21.132240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.112068    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token\") pod \"e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") " pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.172749 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.172702    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:21.487137 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:51:21.487099    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7166b0a0_a0e4_4ee4_b1f1_518bc231f49c.slice/crio-e34d8519a98f3b7b633cea3fb393769c804717a25bee421c87469235ecd1fe40 WatchSource:0}: Error finding container e34d8519a98f3b7b633cea3fb393769c804717a25bee421c87469235ecd1fe40: Status 404 returned error can't find the container with id e34d8519a98f3b7b633cea3fb393769c804717a25bee421c87469235ecd1fe40
Nov 02 17:51:21.488680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.488652    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"]
Nov 02 17:51:21.490169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.490137    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:21.490276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.490213    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:21.490276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.490261    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:21.545478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:21.545447    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acf1cd0-b383-433c-9db9-29accdd26a89" path="/var/lib/kubelet/pods/3acf1cd0-b383-433c-9db9-29accdd26a89/volumes"
Nov 02 17:51:22.028259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:22.028217    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"]
Nov 02 17:51:22.044640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:22.044606    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2502ca91-sd7hn"]
Nov 02 17:51:22.175518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:22.175474    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh" event={"ID":"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c","Type":"ContainerStarted","Data":"ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817"}
Nov 02 17:51:22.175746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:22.175547    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh" event={"ID":"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c","Type":"ContainerStarted","Data":"e34d8519a98f3b7b633cea3fb393769c804717a25bee421c87469235ecd1fe40"}
Nov 02 17:51:22.199081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:22.199014    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh" podStartSLOduration=2.198993409 podStartE2EDuration="2.198993409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:22.197313721 +0000 UTC m=+1094.776723109" watchObservedRunningTime="2026-11-02 17:51:22.198993409 +0000 UTC m=+1094.778402797"
Nov 02 17:51:23.175828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:23.175378    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:23.189477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:23.189113    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:23.196165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:23.196031    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:23.199716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:23.199688    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:51:23.547206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:23.547170    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9250aec-db4b-4371-b805-a6f41d4c80f8" path="/var/lib/kubelet/pods/a9250aec-db4b-4371-b805-a6f41d4c80f8/volumes"
Nov 02 17:51:25.108184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:25.108152    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"]
Nov 02 17:51:25.109038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:25.109004    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" containerID="cri-o://2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60" gracePeriod=30
Nov 02 17:51:28.322744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:28.322700    5042 patch_prober.go:29] interesting pod/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:51:28.323226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:28.322768    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:51:29.064191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:29.064146    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"]
Nov 02 17:51:29.068959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:29.068912    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79886bf2-bjm8l"]
Nov 02 17:51:29.545784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:29.545752    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d9105b-29aa-4587-87d2-72fcfe1dd012" path="/var/lib/kubelet/pods/42d9105b-29aa-4587-87d2-72fcfe1dd012/volumes"
Nov 02 17:51:30.423299 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.423269    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:51:30.477959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.477921    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"]
Nov 02 17:51:30.478945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.478915    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy"
Nov 02 17:51:30.478945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.478943    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy"
Nov 02 17:51:30.479158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.479140    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy"
Nov 02 17:51:30.507864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.486458    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.507864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.497240    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"]
Nov 02 17:51:30.507864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.498609    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6098d0\""
Nov 02 17:51:30.507864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.499308    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6098d0-dockercfg-z9gdh\""
Nov 02 17:51:30.520134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.520109    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"]
Nov 02 17:51:30.521100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.521068    5042 generic.go:356] "Generic (PLEG): container finished" podID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerID="2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60" exitCode=0
Nov 02 17:51:30.521242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.521193    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"
Nov 02 17:51:30.521962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.521921    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy"
Nov 02 17:51:30.521962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.521962    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy"
Nov 02 17:51:30.522205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.522190    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" containerName="istio-proxy"
Nov 02 17:51:30.526377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.526352    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" event={"ID":"c15640b9-72f5-4254-a525-e72f7a75a24e","Type":"ContainerDied","Data":"2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60"}
Nov 02 17:51:30.526479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.526395    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp" event={"ID":"c15640b9-72f5-4254-a525-e72f7a75a24e","Type":"ContainerDied","Data":"b026ccfae00e5f7375628d68c6dc57198378b9142b9ecbec3f9efff76fc5b739"}
Nov 02 17:51:30.526479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.526425    5042 scope.go:122] "RemoveContainer" containerID="2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60"
Nov 02 17:51:30.526479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.526460    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.548253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.548221    5042 scope.go:122] "RemoveContainer" containerID="9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"
Nov 02 17:51:30.552832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.552800    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"]
Nov 02 17:51:30.573857 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.573814    5042 scope.go:122] "RemoveContainer" containerID="2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60"
Nov 02 17:51:30.574404 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:30.574379    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60\": container with ID starting with 2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60 not found: ID does not exist" containerID="2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60"
Nov 02 17:51:30.574559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.574534    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60"} err="failed to get container status \"2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60\": rpc error: code = NotFound desc = could not find container \"2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60\": container with ID starting with 2365428c70e42682b27ef2c7a9e5d5df5f6033e65875711122a0a7178ce6bc60 not found: ID does not exist"
Nov 02 17:51:30.574660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.574648    5042 scope.go:122] "RemoveContainer" containerID="9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"
Nov 02 17:51:30.575459 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:30.575438    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71\": container with ID starting with 9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71 not found: ID does not exist" containerID="9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"
Nov 02 17:51:30.575606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.575583    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71"} err="failed to get container status \"9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71\": rpc error: code = NotFound desc = could not find container \"9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71\": container with ID starting with 9c8775349bb5ea62c2c32e5d1c92814a37ef3ca7571e66188f6cf3c74510fc71 not found: ID does not exist"
Nov 02 17:51:30.618731 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.618688    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.618944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.618758    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.618944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.618796    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.618944 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.618905    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.619127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619004    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.619127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619078    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.619228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619200    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.619287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619261    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.619344 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket\") pod \"c15640b9-72f5-4254-a525-e72f7a75a24e\" (UID: \"c15640b9-72f5-4254-a525-e72f7a75a24e\") "
Nov 02 17:51:30.637087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619573    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.637087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.619902    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.680651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.680567    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:30.682452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.682424    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5dfb\" (UniqueName: \"kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.682975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.682954    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljtc9\" (UniqueName: \"kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9\") pod \"payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.683301 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.683286    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.684497 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.684473    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:30.692058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.692012    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:30.700689 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.700632    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:51:30.712297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.711968    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:30.713098 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.713050    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:30.713312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.713278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:30.731922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.719105    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "kube-api-access-96h9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:30.745878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.743069    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token" pod "c15640b9-72f5-4254-a525-e72f7a75a24e" (UID: "c15640b9-72f5-4254-a525-e72f7a75a24e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.794968    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljtc9\" (UniqueName: \"kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9\") pod \"payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795053    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795227    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795270    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5dfb\" (UniqueName: \"kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795369    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795386    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795402    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795421    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96h9c\" (UniqueName: \"kubernetes.io/projected/c15640b9-72f5-4254-a525-e72f7a75a24e-kube-api-access-96h9c\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795439    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795455    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15640b9-72f5-4254-a525-e72f7a75a24e-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795469    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.795483    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15640b9-72f5-4254-a525-e72f7a75a24e-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.796748    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.797493 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.797408    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.861876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.849066    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5dfb\" (UniqueName: \"kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb\") pod \"payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.861876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.855106    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:30.863904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.863798    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljtc9\" (UniqueName: \"kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9\") pod \"payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.868767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.868743    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:30.904179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.904134    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"]
Nov 02 17:51:30.909260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:30.909227    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c8832b-openshift-default-f9d6b68f7-v85mp"]
Nov 02 17:51:31.174320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.174292    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"]
Nov 02 17:51:31.176214 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:51:31.176185    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd465c3d_0a0a_4622_97d0_5866daa9d3b9.slice/crio-609f98686a3dc2984a892341ffc8a1b046a28a4ca76dacda6e5d5deab342e327 WatchSource:0}: Error finding container 609f98686a3dc2984a892341ffc8a1b046a28a4ca76dacda6e5d5deab342e327: Status 404 returned error can't find the container with id 609f98686a3dc2984a892341ffc8a1b046a28a4ca76dacda6e5d5deab342e327
Nov 02 17:51:31.203456 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:51:31.203428    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode194c4d8_9427_4766_96b4_afce5922a4b5.slice/crio-7a8adf0b9bc98aaf877dff1f405c6d5d01dba8eb9012de099a2ae52eaffbf5f4 WatchSource:0}: Error finding container 7a8adf0b9bc98aaf877dff1f405c6d5d01dba8eb9012de099a2ae52eaffbf5f4: Status 404 returned error can't find the container with id 7a8adf0b9bc98aaf877dff1f405c6d5d01dba8eb9012de099a2ae52eaffbf5f4
Nov 02 17:51:31.206011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.205987    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"]
Nov 02 17:51:31.532656 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.532562    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n" event={"ID":"e194c4d8-9427-4766-96b4-afce5922a4b5","Type":"ContainerStarted","Data":"7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47"}
Nov 02 17:51:31.532854 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.532665    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n" event={"ID":"e194c4d8-9427-4766-96b4-afce5922a4b5","Type":"ContainerStarted","Data":"7a8adf0b9bc98aaf877dff1f405c6d5d01dba8eb9012de099a2ae52eaffbf5f4"}
Nov 02 17:51:31.532854 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.532725    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:31.534674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.534639    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56" event={"ID":"fd465c3d-0a0a-4622-97d0-5866daa9d3b9","Type":"ContainerStarted","Data":"6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e"}
Nov 02 17:51:31.534811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.534691    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56" event={"ID":"fd465c3d-0a0a-4622-97d0-5866daa9d3b9","Type":"ContainerStarted","Data":"609f98686a3dc2984a892341ffc8a1b046a28a4ca76dacda6e5d5deab342e327"}
Nov 02 17:51:31.534811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.534806    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:31.545675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.545628    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15640b9-72f5-4254-a525-e72f7a75a24e" path="/var/lib/kubelet/pods/c15640b9-72f5-4254-a525-e72f7a75a24e/volumes"
Nov 02 17:51:31.571668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.571614    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n" podStartSLOduration=1.5715996840000002 podStartE2EDuration="1.571599684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:31.570144221 +0000 UTC m=+1104.149553646" watchObservedRunningTime="2026-11-02 17:51:31.571599684 +0000 UTC m=+1104.151009070"
Nov 02 17:51:31.609313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:31.604131    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56" podStartSLOduration=1.604109737 podStartE2EDuration="1.604109737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:31.602590168 +0000 UTC m=+1104.181999567" watchObservedRunningTime="2026-11-02 17:51:31.604109737 +0000 UTC m=+1104.183519124"
Nov 02 17:51:32.485309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.485269    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"]
Nov 02 17:51:32.497204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.491951    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.502941 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.502908    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"]
Nov 02 17:51:32.563219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.563181    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7dbdb8-dockercfg-xp2w2\""
Nov 02 17:51:32.563533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.563515    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7dbdb8\""
Nov 02 17:51:32.658913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.658872    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.684183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.684150    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldg5\" (UniqueName: \"kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5\") pod \"payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.790901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.788110    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.790901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.788222    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldg5\" (UniqueName: \"kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5\") pod \"payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.790901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.789429    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.798708 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.798505    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldg5\" (UniqueName: \"kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5\") pod \"payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:32.819228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:32.819192    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:33.123878 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:51:33.123815    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cc72d0_0825_4365_bf3f_50dfaccc32dd.slice/crio-9f8acedd7ca70f7b33863e13fbaea2c944a5d53a06e3f75188186e4c24a197d6 WatchSource:0}: Error finding container 9f8acedd7ca70f7b33863e13fbaea2c944a5d53a06e3f75188186e4c24a197d6: Status 404 returned error can't find the container with id 9f8acedd7ca70f7b33863e13fbaea2c944a5d53a06e3f75188186e4c24a197d6
Nov 02 17:51:33.124710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:33.124682    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"]
Nov 02 17:51:33.556751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:33.556718    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w" event={"ID":"03cc72d0-0825-4365-bf3f-50dfaccc32dd","Type":"ContainerStarted","Data":"fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250"}
Nov 02 17:51:33.557024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:33.556771    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w" event={"ID":"03cc72d0-0825-4365-bf3f-50dfaccc32dd","Type":"ContainerStarted","Data":"9f8acedd7ca70f7b33863e13fbaea2c944a5d53a06e3f75188186e4c24a197d6"}
Nov 02 17:51:33.557024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:33.556815    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:33.588073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:33.588008    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w" podStartSLOduration=1.587988513 podStartE2EDuration="1.587988513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:33.587052052 +0000 UTC m=+1106.166461439" watchObservedRunningTime="2026-11-02 17:51:33.587988513 +0000 UTC m=+1106.167397898"
Nov 02 17:51:38.049681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:38.049645    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"]
Nov 02 17:51:38.054553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:38.054526    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01287c72-mz8b6"]
Nov 02 17:51:38.315216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:38.315114    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"]
Nov 02 17:51:38.334857 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:38.334808    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"]
Nov 02 17:51:39.544675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:39.544638    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6642c09e-eabf-4215-85ed-32fff0324e5d" path="/var/lib/kubelet/pods/6642c09e-eabf-4215-85ed-32fff0324e5d/volumes"
Nov 02 17:51:42.551484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:42.551457    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:42.552275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:42.552253    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:43.316235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.316171    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc" podUID="62215556-fff0-4c26-a7d3-681dbe5a6d1c" containerName="payload-processing" containerID="cri-o://811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157" gracePeriod=25
Nov 02 17:51:43.335398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.335344    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg" podUID="04657e80-f3ed-443a-986f-7ae228f4a891" containerName="payload-pre-processing" containerID="cri-o://14c128ad0479b94c27a52751b1f61d6a33f61cf46a2d76e80cc4cdf8148abfee" gracePeriod=25
Nov 02 17:51:43.768778 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.768738    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:51:43.827385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.827356    5042 generic.go:356] "Generic (PLEG): container finished" podID="62215556-fff0-4c26-a7d3-681dbe5a6d1c" containerID="811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157" exitCode=0
Nov 02 17:51:43.827649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.827630    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc" event={"ID":"62215556-fff0-4c26-a7d3-681dbe5a6d1c","Type":"ContainerDied","Data":"811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157"}
Nov 02 17:51:43.827791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.827775    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc" event={"ID":"62215556-fff0-4c26-a7d3-681dbe5a6d1c","Type":"ContainerDied","Data":"e59d92b6f9df58e2a20186a11a98f026d435723624768ee99f0ffb0d74652b65"}
Nov 02 17:51:43.829898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.828973    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"
Nov 02 17:51:43.830580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.828989    5042 scope.go:122] "RemoveContainer" containerID="811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157"
Nov 02 17:51:43.833305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.833045    5042 generic.go:356] "Generic (PLEG): container finished" podID="04657e80-f3ed-443a-986f-7ae228f4a891" containerID="14c128ad0479b94c27a52751b1f61d6a33f61cf46a2d76e80cc4cdf8148abfee" exitCode=0
Nov 02 17:51:43.833305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.833260    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg" event={"ID":"04657e80-f3ed-443a-986f-7ae228f4a891","Type":"ContainerDied","Data":"14c128ad0479b94c27a52751b1f61d6a33f61cf46a2d76e80cc4cdf8148abfee"}
Nov 02 17:51:43.856880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.855429    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume\") pod \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") "
Nov 02 17:51:43.856880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.855485    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm\" (UniqueName: \"kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm\") pod \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\" (UID: \"62215556-fff0-4c26-a7d3-681dbe5a6d1c\") "
Nov 02 17:51:43.856880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.856759    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume" pod "62215556-fff0-4c26-a7d3-681dbe5a6d1c" (UID: "62215556-fff0-4c26-a7d3-681dbe5a6d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:43.859230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.859201    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm" pod "62215556-fff0-4c26-a7d3-681dbe5a6d1c" (UID: "62215556-fff0-4c26-a7d3-681dbe5a6d1c"). InnerVolumeSpecName "kube-api-access-9dvdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:43.869513 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.869487    5042 scope.go:122] "RemoveContainer" containerID="811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157"
Nov 02 17:51:43.870391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.870368    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:51:43.870935 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:43.870909    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157\": container with ID starting with 811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157 not found: ID does not exist" containerID="811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157"
Nov 02 17:51:43.870979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.870951    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157"} err="failed to get container status \"811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157\": rpc error: code = NotFound desc = could not find container \"811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157\": container with ID starting with 811a01e6179ee5ea23b631fc785c68283ec96ebc56ff60b12927f24eab67e157 not found: ID does not exist"
Nov 02 17:51:43.957355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.957319    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8\" (UniqueName: \"kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8\") pod \"04657e80-f3ed-443a-986f-7ae228f4a891\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") "
Nov 02 17:51:43.957564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.957542    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume\") pod \"04657e80-f3ed-443a-986f-7ae228f4a891\" (UID: \"04657e80-f3ed-443a-986f-7ae228f4a891\") "
Nov 02 17:51:43.958344 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.958312    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62215556-fff0-4c26-a7d3-681dbe5a6d1c-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:43.958344 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.958346    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvdm\" (UniqueName: \"kubernetes.io/projected/62215556-fff0-4c26-a7d3-681dbe5a6d1c-kube-api-access-9dvdm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:43.958750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.958716    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume" pod "04657e80-f3ed-443a-986f-7ae228f4a891" (UID: "04657e80-f3ed-443a-986f-7ae228f4a891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:43.961628 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:43.961598    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8" pod "04657e80-f3ed-443a-986f-7ae228f4a891" (UID: "04657e80-f3ed-443a-986f-7ae228f4a891"). InnerVolumeSpecName "kube-api-access-lxqb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:44.059560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.059508    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04657e80-f3ed-443a-986f-7ae228f4a891-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:44.059560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.059549    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqb8\" (UniqueName: \"kubernetes.io/projected/04657e80-f3ed-443a-986f-7ae228f4a891-kube-api-access-lxqb8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:51:44.165943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.165909    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"]
Nov 02 17:51:44.169478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.169430    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cd2738-6789f47f85-7dvjc"]
Nov 02 17:51:44.604161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.604125    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:51:44.852790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.852763    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"
Nov 02 17:51:44.853239 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.852794    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg" event={"ID":"04657e80-f3ed-443a-986f-7ae228f4a891","Type":"ContainerDied","Data":"c76d0ecbe0c37f7f984ee48244f1070f2b7c98414f563d86d936d7b5ebd10369"}
Nov 02 17:51:44.853239 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.852867    5042 scope.go:122] "RemoveContainer" containerID="14c128ad0479b94c27a52751b1f61d6a33f61cf46a2d76e80cc4cdf8148abfee"
Nov 02 17:51:44.912973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.912915    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"]
Nov 02 17:51:44.917464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:44.917435    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cd2738-6c7bcfd6c5-bw2fg"]
Nov 02 17:51:45.544756 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:45.544725    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04657e80-f3ed-443a-986f-7ae228f4a891" path="/var/lib/kubelet/pods/04657e80-f3ed-443a-986f-7ae228f4a891/volumes"
Nov 02 17:51:45.545237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:45.545223    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62215556-fff0-4c26-a7d3-681dbe5a6d1c" path="/var/lib/kubelet/pods/62215556-fff0-4c26-a7d3-681dbe5a6d1c/volumes"
Nov 02 17:51:48.614925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:48.614686    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5_98e1b231-b167-4dc3-8820-0126f6a8d008/payload-processing/0.log"
Nov 02 17:51:48.732872 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:51:48.731094    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:51:49.219492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:49.219462    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj_0869f30c-b58a-40ee-9108-d4c504934fd7/payload-processing/0.log"
Nov 02 17:51:49.411148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:49.411066    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj_0869f30c-b58a-40ee-9108-d4c504934fd7/payload-processing/0.log"
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.570049    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"]
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.570913    5042 cpu_manager.go:414] "containerMap: removing container" podUID="62215556-fff0-4c26-a7d3-681dbe5a6d1c" containerName="payload-processing"
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.570929    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62215556-fff0-4c26-a7d3-681dbe5a6d1c" containerName="payload-processing"
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.570950    5042 cpu_manager.go:414] "containerMap: removing container" podUID="04657e80-f3ed-443a-986f-7ae228f4a891" containerName="payload-pre-processing"
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.570957    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04657e80-f3ed-443a-986f-7ae228f4a891" containerName="payload-pre-processing"
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.571161    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="62215556-fff0-4c26-a7d3-681dbe5a6d1c" containerName="payload-processing"
Nov 02 17:51:51.573024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.571171    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="04657e80-f3ed-443a-986f-7ae228f4a891" containerName="payload-pre-processing"
Nov 02 17:51:51.581135 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.581113    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:51.701782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.662191    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-009a5f-openshift-default-dockercfg-pq6b9\""
Nov 02 17:51:51.968396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.852664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:51.968396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.852762    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:51.968396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.852972    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:51.968396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.853114    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:51.992685 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:51.992649    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.007316 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.007289    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.007646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.007626    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.007804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.007789    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcpg\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.008081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.008066    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.008210 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.008197    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.008479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.008463    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.008652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.008609    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.008829 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.008799    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.009521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.009501    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.009975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.009955    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.010382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.010362    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.032466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.032292    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"]
Nov 02 17:51:52.064863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.063247    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.111490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.111454    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.111828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.111810    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcpg\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.112068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.112051    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.112266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.112251    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.112383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.112370    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.113479 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.113454    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.186118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.186087    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.221405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.220534    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.233595 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.233362    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcpg\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.252903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.252526    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token\") pod \"e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") " pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.382794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.382503    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:52.757290 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:51:52.757240    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffa6cd4_155d_47ff_8d38_420db30b67cd.slice/crio-45ddd9e3f63d5804d9d7ab974a6884612f84ec4cd6947c6d90cebed9da28efb0 WatchSource:0}: Error finding container 45ddd9e3f63d5804d9d7ab974a6884612f84ec4cd6947c6d90cebed9da28efb0: Status 404 returned error can't find the container with id 45ddd9e3f63d5804d9d7ab974a6884612f84ec4cd6947c6d90cebed9da28efb0
Nov 02 17:51:52.758596 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.758566    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"]
Nov 02 17:51:52.760282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.760244    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:52.760384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.760317    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:52.760384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:52.760352    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:53.019409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:53.019320    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" event={"ID":"8ffa6cd4-155d-47ff-8d38-420db30b67cd","Type":"ContainerStarted","Data":"eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"}
Nov 02 17:51:53.019409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:53.019392    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" event={"ID":"8ffa6cd4-155d-47ff-8d38-420db30b67cd","Type":"ContainerStarted","Data":"45ddd9e3f63d5804d9d7ab974a6884612f84ec4cd6947c6d90cebed9da28efb0"}
Nov 02 17:51:53.068631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:53.068566    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podStartSLOduration=2.068545745 podStartE2EDuration="2.068545745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:53.063711662 +0000 UTC m=+1125.643121048" watchObservedRunningTime="2026-11-02 17:51:53.068545745 +0000 UTC m=+1125.647955112"
Nov 02 17:51:53.384901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:53.384796    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:53.390382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:53.390350    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:54.028502 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:54.028467    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:54.029515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:54.029495    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:51:54.215714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:54.215668    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"]
Nov 02 17:51:54.223112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:54.222825    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"]
Nov 02 17:51:55.038956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:55.038923    5042 generic.go:356] "Generic (PLEG): container finished" podID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerID="ca5773e640611b6702517d50f03098feeeeae145efc737b4cd5620a0488d2063" exitCode=0
Nov 02 17:51:55.039465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:55.038998    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" event={"ID":"14ba8a64-73a1-4122-b4f0-d3b1579aaad4","Type":"ContainerDied","Data":"ca5773e640611b6702517d50f03098feeeeae145efc737b4cd5620a0488d2063"}
Nov 02 17:51:55.039786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:55.039767    5042 scope.go:122] "RemoveContainer" containerID="ca5773e640611b6702517d50f03098feeeeae145efc737b4cd5620a0488d2063"
Nov 02 17:51:55.041379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:55.041347    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:55.041490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:55.041439    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:55.041558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:55.041487    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:56.050213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:56.050174    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" event={"ID":"14ba8a64-73a1-4122-b4f0-d3b1579aaad4","Type":"ContainerStarted","Data":"d353e3f69ad6b00f86a7f4beaeca6aa2c0982e9f32ef564315290cccaeeb69ad"}
Nov 02 17:51:56.226965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:56.226933    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:51:56.227243 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:56.227203    5042 patch_prober.go:29] interesting pod/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 17:51:56.227306 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:56.227274    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 17:51:57.200864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.097744    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"]
Nov 02 17:51:57.357028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.356995    5042 patch_prober.go:29] interesting pod/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 17:51:57.357301 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.357268    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 17:51:57.380046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.379995    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"]
Nov 02 17:51:57.395983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.390531    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.447388 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.447290    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.447659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.447637    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.447787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.447772    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.447938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.447923    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.448090 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.448077    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.448215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.448199    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.448330 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.448316    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.448511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.448494    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqvk\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.448717 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.448702    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.453881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.453119    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4c9298-openshift-default-dockercfg-rqzvt\""
Nov 02 17:51:57.550202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550172    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550211    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550243    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550280    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550326    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550366    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550401    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550475    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqvk\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550531    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.550746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550697    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.551005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550980    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.551005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.550999    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.551694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.551668    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.552296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.552275    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.553460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.553435    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.553645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.553570    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.566416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.566383    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.567499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.567472    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqvk\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk\") pod \"e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") " pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:57.776470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:57.776125    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:51:58.291143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:58.262228    5042 patch_prober.go:29] interesting pod/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 17:51:58.291143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:58.262280    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 17:51:58.332645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:58.332494    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"]
Nov 02 17:51:58.369413 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:51:58.369350    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a77187_c64b_4931_8aa4_4b416902f804.slice/crio-ab03578e339cf55c77b0eb1cb907635c71b1f42606afb31d903b68477a7f72c3 WatchSource:0}: Error finding container ab03578e339cf55c77b0eb1cb907635c71b1f42606afb31d903b68477a7f72c3: Status 404 returned error can't find the container with id ab03578e339cf55c77b0eb1cb907635c71b1f42606afb31d903b68477a7f72c3
Nov 02 17:51:58.432187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:58.431809    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:58.432187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:58.431969    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:58.432187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:58.432012    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:59.217139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.216567    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56" podUID="fd465c3d-0a0a-4622-97d0-5866daa9d3b9" containerName="payload-pre-processing" containerID="cri-o://6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e" gracePeriod=25
Nov 02 17:51:59.225297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.225031    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n" podUID="e194c4d8-9427-4766-96b4-afce5922a4b5" containerName="payload-processing" containerID="cri-o://7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47" gracePeriod=25
Nov 02 17:51:59.227337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.227308    5042 patch_prober.go:29] interesting pod/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 17:51:59.227564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.227526    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 17:51:59.748585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.747128    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:59.753596 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.753574    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:59.947297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.946240    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb\" (UniqueName: \"kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb\") pod \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") "
Nov 02 17:51:59.947297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.946339    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume\") pod \"e194c4d8-9427-4766-96b4-afce5922a4b5\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") "
Nov 02 17:51:59.947297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.946454    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume\") pod \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\" (UID: \"fd465c3d-0a0a-4622-97d0-5866daa9d3b9\") "
Nov 02 17:51:59.947297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.946564    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9\" (UniqueName: \"kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9\") pod \"e194c4d8-9427-4766-96b4-afce5922a4b5\" (UID: \"e194c4d8-9427-4766-96b4-afce5922a4b5\") "
Nov 02 17:51:59.948715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.948686    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume" pod "e194c4d8-9427-4766-96b4-afce5922a4b5" (UID: "e194c4d8-9427-4766-96b4-afce5922a4b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:59.950510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.950481    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume" pod "fd465c3d-0a0a-4622-97d0-5866daa9d3b9" (UID: "fd465c3d-0a0a-4622-97d0-5866daa9d3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:59.951055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.951029    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9" pod "e194c4d8-9427-4766-96b4-afce5922a4b5" (UID: "e194c4d8-9427-4766-96b4-afce5922a4b5"). InnerVolumeSpecName "kube-api-access-ljtc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:59.952718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.952685    5042 generic.go:356] "Generic (PLEG): container finished" podID="e194c4d8-9427-4766-96b4-afce5922a4b5" containerID="7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47" exitCode=0
Nov 02 17:51:59.952966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.952944    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n" event={"ID":"e194c4d8-9427-4766-96b4-afce5922a4b5","Type":"ContainerDied","Data":"7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47"}
Nov 02 17:51:59.953171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.953136    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n" event={"ID":"e194c4d8-9427-4766-96b4-afce5922a4b5","Type":"ContainerDied","Data":"7a8adf0b9bc98aaf877dff1f405c6d5d01dba8eb9012de099a2ae52eaffbf5f4"}
Nov 02 17:51:59.953320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.953248    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb" pod "fd465c3d-0a0a-4622-97d0-5866daa9d3b9" (UID: "fd465c3d-0a0a-4622-97d0-5866daa9d3b9"). InnerVolumeSpecName "kube-api-access-n5dfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:59.953511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.953329    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"
Nov 02 17:51:59.954069 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.953340    5042 scope.go:122] "RemoveContainer" containerID="7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47"
Nov 02 17:51:59.957262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.957231    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" event={"ID":"04a77187-c64b-4931-8aa4-4b416902f804","Type":"ContainerStarted","Data":"265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63"}
Nov 02 17:51:59.957378 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.957295    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" event={"ID":"04a77187-c64b-4931-8aa4-4b416902f804","Type":"ContainerStarted","Data":"ab03578e339cf55c77b0eb1cb907635c71b1f42606afb31d903b68477a7f72c3"}
Nov 02 17:51:59.963396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.963361    5042 generic.go:356] "Generic (PLEG): container finished" podID="fd465c3d-0a0a-4622-97d0-5866daa9d3b9" containerID="6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e" exitCode=0
Nov 02 17:51:59.963526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.963508    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56" event={"ID":"fd465c3d-0a0a-4622-97d0-5866daa9d3b9","Type":"ContainerDied","Data":"6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e"}
Nov 02 17:51:59.963595 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.963553    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56" event={"ID":"fd465c3d-0a0a-4622-97d0-5866daa9d3b9","Type":"ContainerDied","Data":"609f98686a3dc2984a892341ffc8a1b046a28a4ca76dacda6e5d5deab342e327"}
Nov 02 17:51:59.963650 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.963630    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"
Nov 02 17:51:59.997138 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:51:59.993658    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podStartSLOduration=2.9936373769999998 podStartE2EDuration="2.993637377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:59.992887467 +0000 UTC m=+1132.572296855" watchObservedRunningTime="2026-11-02 17:51:59.993637377 +0000 UTC m=+1132.573046765"
Nov 02 17:52:00.017787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.017762    5042 scope.go:122] "RemoveContainer" containerID="7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47"
Nov 02 17:52:00.020240 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:00.020186    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47\": container with ID starting with 7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47 not found: ID does not exist" containerID="7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47"
Nov 02 17:52:00.020378 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.020232    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47"} err="failed to get container status \"7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47\": rpc error: code = NotFound desc = could not find container \"7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47\": container with ID starting with 7e7c70386e6bb6ed2361b58550731178b8a382cdc1312e1f1b049682f1f26a47 not found: ID does not exist"
Nov 02 17:52:00.020378 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.020260    5042 scope.go:122] "RemoveContainer" containerID="6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e"
Nov 02 17:52:00.030384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.030335    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"]
Nov 02 17:52:00.035311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.035290    5042 scope.go:122] "RemoveContainer" containerID="6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e"
Nov 02 17:52:00.035704 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:00.035681    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e\": container with ID starting with 6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e not found: ID does not exist" containerID="6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e"
Nov 02 17:52:00.035745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.035717    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e"} err="failed to get container status \"6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e\": rpc error: code = NotFound desc = could not find container \"6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e\": container with ID starting with 6a6f7a6ace3a9fed78cc7949905d0dc9c0acd4c109f5d953354716e15c80452e not found: ID does not exist"
Nov 02 17:52:00.036340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.036322    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6098d0-57f8b4f46c-cvn56"]
Nov 02 17:52:00.049119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.049089    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"]
Nov 02 17:52:00.049911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.049885    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5dfb\" (UniqueName: \"kubernetes.io/projected/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-kube-api-access-n5dfb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:00.050097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.050079    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e194c4d8-9427-4766-96b4-afce5922a4b5-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:00.050557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.050212    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd465c3d-0a0a-4622-97d0-5866daa9d3b9-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:00.050557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.050232    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljtc9\" (UniqueName: \"kubernetes.io/projected/e194c4d8-9427-4766-96b4-afce5922a4b5-kube-api-access-ljtc9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:00.054457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.054434    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6098d0-6b6bb6fb87-dgh9n"]
Nov 02 17:52:00.227292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.227257    5042 patch_prober.go:29] interesting pod/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 17:52:00.227477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.227354    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 17:52:00.786867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.781740    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:52:00.786867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.782047    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 17:52:00.786867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:00.782096    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 17:52:01.234865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:01.234278    5042 patch_prober.go:29] interesting pod/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Nov 02 17:52:01.234865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:01.234342    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Nov 02 17:52:01.546218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:01.546142    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e194c4d8-9427-4766-96b4-afce5922a4b5" path="/var/lib/kubelet/pods/e194c4d8-9427-4766-96b4-afce5922a4b5/volumes"
Nov 02 17:52:01.546645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:01.546630    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd465c3d-0a0a-4622-97d0-5866daa9d3b9" path="/var/lib/kubelet/pods/fd465c3d-0a0a-4622-97d0-5866daa9d3b9/volumes"
Nov 02 17:52:01.799536 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:01.799447    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 17:52:01.800105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:01.800070    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 17:52:02.238538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:02.237173    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:52:02.238538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:02.237775    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:52:02.238793 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:02.238597    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:52:02.791865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:02.789773    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 17:52:02.791865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:02.789867    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 17:52:03.738331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.738292    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"]
Nov 02 17:52:03.739348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.739320    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e194c4d8-9427-4766-96b4-afce5922a4b5" containerName="payload-processing"
Nov 02 17:52:03.739348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.739348    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e194c4d8-9427-4766-96b4-afce5922a4b5" containerName="payload-processing"
Nov 02 17:52:03.739535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.739375    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fd465c3d-0a0a-4622-97d0-5866daa9d3b9" containerName="payload-pre-processing"
Nov 02 17:52:03.739535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.739383    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd465c3d-0a0a-4622-97d0-5866daa9d3b9" containerName="payload-pre-processing"
Nov 02 17:52:03.739619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.739601    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e194c4d8-9427-4766-96b4-afce5922a4b5" containerName="payload-processing"
Nov 02 17:52:03.739619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.739614    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd465c3d-0a0a-4622-97d0-5866daa9d3b9" containerName="payload-pre-processing"
Nov 02 17:52:03.745670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.745650    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:03.752480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.752451    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"]
Nov 02 17:52:03.787864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.786421    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4a5f09-dockercfg-4l2tn\""
Nov 02 17:52:03.787864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.786513    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4a5f09\""
Nov 02 17:52:03.787864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.786587    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 17:52:03.787864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.786625    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 17:52:03.874866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.872937    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ccm\" (UniqueName: \"kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm\") pod \"payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:03.874866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.873199    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:03.974902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.974826    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ccm\" (UniqueName: \"kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm\") pod \"payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:03.975149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.975130    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:03.976169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:03.976138    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:04.155075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:04.154983    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ccm\" (UniqueName: \"kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm\") pod \"payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:04.363138 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:04.363000    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:04.826467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:04.818686    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 17:52:04.826467 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:04.818749    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 17:52:04.950584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:04.949944    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"]
Nov 02 17:52:05.313861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:05.301606    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm" event={"ID":"095bbff2-398e-4520-9e8a-2fe5355ed7cc","Type":"ContainerStarted","Data":"4c663d272fb62f10b3c174c0cebadd744220a27aed4a3242db14d0a97e8ca052"}
Nov 02 17:52:05.785021 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:05.784925    5042 patch_prober.go:29] interesting pod/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Nov 02 17:52:05.785337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:05.785294    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Nov 02 17:52:06.314786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:06.314745    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm" event={"ID":"095bbff2-398e-4520-9e8a-2fe5355ed7cc","Type":"ContainerStarted","Data":"a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d"}
Nov 02 17:52:06.315308 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:06.314851    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:06.349968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:06.349916    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm" podStartSLOduration=3.349902252 podStartE2EDuration="3.349902252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:06.349282509 +0000 UTC m=+1138.928691897" watchObservedRunningTime="2026-11-02 17:52:06.349902252 +0000 UTC m=+1138.929311634"
Nov 02 17:52:06.872472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:06.872430    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:52:06.874112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:06.874085    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:52:06.875380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:06.875356    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:52:09.399733 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.399692    5042 scope.go:122] "RemoveContainer" containerID="6433d99ef8a2e5272db048edab7d2fdcfde36ca7a2e7a1f152bd414c900811ed"
Nov 02 17:52:09.412785 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.412760    5042 scope.go:122] "RemoveContainer" containerID="69de85df32fca7d9af21bffd57505e7b1b4d25e88a177154ad8e8991c8b891ba"
Nov 02 17:52:09.426256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.426230    5042 scope.go:122] "RemoveContainer" containerID="2acc4285daac96a7cdf4e38e9b3c29f910797d43b8851673f74fe47721a4f696"
Nov 02 17:52:09.456827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.454722    5042 scope.go:122] "RemoveContainer" containerID="c33de816728db41e86a96ecb9b1009a19bc7df9ca39f2c342d2646433516fc94"
Nov 02 17:52:09.516483 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.516451    5042 scope.go:122] "RemoveContainer" containerID="a03154a250270a386f4d1086cd14f65488c8c55fd352972ec52036b6a56c2652"
Nov 02 17:52:09.555074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.554933    5042 scope.go:122] "RemoveContainer" containerID="b4a6ca67d24954686d027fcc4afe42a1064592ad847ba8e0d7ad6ad2498de863"
Nov 02 17:52:09.575333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.575308    5042 scope.go:122] "RemoveContainer" containerID="3cd41ca23cc563343ec1d099276b6df29edf5a2ebc31f4027ae38970bf8263ed"
Nov 02 17:52:09.587138 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.587112    5042 scope.go:122] "RemoveContainer" containerID="ceb549d576e7c0f4ddca390d87fb3e77ff333fb897b28eb1bbb087f964f14087"
Nov 02 17:52:09.606046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.606017    5042 scope.go:122] "RemoveContainer" containerID="29be0999b77d1b7e8aa0983860a1f96dd2b656de5be9bee48fa0afb59329eec9"
Nov 02 17:52:09.618499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:09.618475    5042 scope.go:122] "RemoveContainer" containerID="1f159dd9ab9896a01115ad823eb887b6414fa54ba57232e704f8110a82462ca2"
Nov 02 17:52:16.528315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:16.528278    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"]
Nov 02 17:52:16.528951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:16.528507    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy" containerID="cri-o://292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6" gracePeriod=30
Nov 02 17:52:17.349656 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:17.349623    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:52:18.542649 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:18.542611    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:52:21.038914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:21.036918    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"]
Nov 02 17:52:21.069864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:21.066959    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c2dba23-5k6mf"]
Nov 02 17:52:21.549825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:21.549790    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fa5824-f698-4ddc-8f8e-5620d98b0c8d" path="/var/lib/kubelet/pods/f7fa5824-f698-4ddc-8f8e-5620d98b0c8d/volumes"
Nov 02 17:52:22.133040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.133015    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210406    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210533    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210609    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210687    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210773    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210892    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.210933    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.211032    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.211065    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token\") pod \"21b74354-9f55-4fc1-b42b-496a5838f504\" (UID: \"21b74354-9f55-4fc1-b42b-496a5838f504\") "
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.212422    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.212741    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:22.213813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.213001    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:22.214619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.214335    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:22.214619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.214404    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:22.241220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.239167    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:22.251891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.244684    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:22.251891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.245322    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "kube-api-access-6qkf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:22.261509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.261475    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token" pod "21b74354-9f55-4fc1-b42b-496a5838f504" (UID: "21b74354-9f55-4fc1-b42b-496a5838f504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:22.313645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313598    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b74354-9f55-4fc1-b42b-496a5838f504-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.313645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313632    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.313645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313641    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.313645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313654    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qkf8\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-kube-api-access-6qkf8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.314047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313664    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b74354-9f55-4fc1-b42b-496a5838f504-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.314047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313672    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b74354-9f55-4fc1-b42b-496a5838f504-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.314047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313682    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.314047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313691    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.314047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.313700    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b74354-9f55-4fc1-b42b-496a5838f504-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:22.559029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.558677    5042 generic.go:356] "Generic (PLEG): container finished" podID="21b74354-9f55-4fc1-b42b-496a5838f504" containerID="292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6" exitCode=0
Nov 02 17:52:22.559029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.558985    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"
Nov 02 17:52:22.566385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.563900    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" event={"ID":"21b74354-9f55-4fc1-b42b-496a5838f504","Type":"ContainerDied","Data":"292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6"}
Nov 02 17:52:22.566385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.564028    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks" event={"ID":"21b74354-9f55-4fc1-b42b-496a5838f504","Type":"ContainerDied","Data":"c703090d7024a0c0a95b86846b47041db7feafc52955ec2928fd5474d1bcd693"}
Nov 02 17:52:22.566385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.564066    5042 scope.go:122] "RemoveContainer" containerID="292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6"
Nov 02 17:52:22.631596 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.631156    5042 scope.go:122] "RemoveContainer" containerID="292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6"
Nov 02 17:52:22.632511 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:22.632407    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6\": container with ID starting with 292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6 not found: ID does not exist" containerID="292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6"
Nov 02 17:52:22.632511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.632449    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6"} err="failed to get container status \"292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6\": rpc error: code = NotFound desc = could not find container \"292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6\": container with ID starting with 292ddbbd2232d26bfa3f40b60e17e3a35e7e7fbecc8be8d3c9fdb862c81ec4b6 not found: ID does not exist"
Nov 02 17:52:22.674189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.674150    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"]
Nov 02 17:52:22.699379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:22.699343    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-lg9ks"]
Nov 02 17:52:23.695672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:23.644825    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" path="/var/lib/kubelet/pods/21b74354-9f55-4fc1-b42b-496a5838f504/volumes"
Nov 02 17:52:25.919796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:25.919714    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"]
Nov 02 17:52:25.920660 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:25.920630    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy" containerID="cri-o://d353e3f69ad6b00f86a7f4beaeca6aa2c0982e9f32ef564315290cccaeeb69ad" gracePeriod=30
Nov 02 17:52:28.017392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.017344    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"]
Nov 02 17:52:28.018454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.018345    5042 cpu_manager.go:414] "containerMap: removing container" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy"
Nov 02 17:52:28.018454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.018367    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy"
Nov 02 17:52:28.018651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.018570    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b74354-9f55-4fc1-b42b-496a5838f504" containerName="istio-proxy"
Nov 02 17:52:28.024338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.024315    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.029898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.029860    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"]
Nov 02 17:52:28.034780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.034754    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d8e217-openshift-default-dockercfg-znfmz\""
Nov 02 17:52:28.115868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.115439    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.129861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.116500    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.129861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.117340    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.143186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.143146    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.144758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.144727    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.145041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.145020    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.145222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.145199    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.145410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.145392    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.145607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.145591    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252047    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252160    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252206    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252236    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252342    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252408    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252475    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252520    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.252917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.252594    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.319222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.253756    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.319222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.254072    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.396949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.372883    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.396949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.373133    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.396949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.373582    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.403896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.397990    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.425865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.425302    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.425865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.425857    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.435628 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.435056    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5\") pod \"e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") " pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:28.654945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:28.652321    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:29.165340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:29.154635    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"]
Nov 02 17:52:29.169643 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:52:29.169576    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c8f333_4d05_40fa_a759_543d212b79e2.slice/crio-d31f400c4e4b4c4561c1b1b6a6e91b9356d2eb5aee8787a20f73330eb58fdd41 WatchSource:0}: Error finding container d31f400c4e4b4c4561c1b1b6a6e91b9356d2eb5aee8787a20f73330eb58fdd41: Status 404 returned error can't find the container with id d31f400c4e4b4c4561c1b1b6a6e91b9356d2eb5aee8787a20f73330eb58fdd41
Nov 02 17:52:29.204289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:29.204230    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:29.204504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:29.204320    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:29.204504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:29.204354    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:30.149352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:30.149272    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" event={"ID":"41c8f333-4d05-40fa-a759-543d212b79e2","Type":"ContainerStarted","Data":"fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"}
Nov 02 17:52:30.149352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:30.149332    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" event={"ID":"41c8f333-4d05-40fa-a759-543d212b79e2","Type":"ContainerStarted","Data":"d31f400c4e4b4c4561c1b1b6a6e91b9356d2eb5aee8787a20f73330eb58fdd41"}
Nov 02 17:52:30.173368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:30.173306    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podStartSLOduration=3.173291325 podStartE2EDuration="3.173291325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:30.171025362 +0000 UTC m=+1162.750434749" watchObservedRunningTime="2026-11-02 17:52:30.173291325 +0000 UTC m=+1162.752700712"
Nov 02 17:52:30.663087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:30.660960    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:30.688041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:30.687994    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:31.170864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.167458    5042 generic.go:356] "Generic (PLEG): container finished" podID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerID="d353e3f69ad6b00f86a7f4beaeca6aa2c0982e9f32ef564315290cccaeeb69ad" exitCode=0
Nov 02 17:52:31.170864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.169895    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" event={"ID":"14ba8a64-73a1-4122-b4f0-d3b1579aaad4","Type":"ContainerDied","Data":"d353e3f69ad6b00f86a7f4beaeca6aa2c0982e9f32ef564315290cccaeeb69ad"}
Nov 02 17:52:31.170864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.169978    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:31.170864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.170012    5042 scope.go:122] "RemoveContainer" containerID="ca5773e640611b6702517d50f03098feeeeae145efc737b4cd5620a0488d2063"
Nov 02 17:52:31.171223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.171148    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:52:31.240863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.240376    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:52:31.310128 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.310093    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.310463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.310442    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.310605 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.310590    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.310727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.310712    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.310868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.310830    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.311105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.311090    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.311233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.311219    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.311339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.311326    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.311514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.311501    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo\") pod \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\" (UID: \"14ba8a64-73a1-4122-b4f0-d3b1579aaad4\") "
Nov 02 17:52:31.315917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.315440    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:31.315917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.315765    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:31.315917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.315827    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:31.315917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.315894    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:31.316520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.316499    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:31.320006 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.319949    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:31.329824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.329752    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "kube-api-access-vwqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:31.330291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.330237    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:31.333095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.333036    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy" pod "14ba8a64-73a1-4122-b4f0-d3b1579aaad4" (UID: "14ba8a64-73a1-4122-b4f0-d3b1579aaad4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:31.415338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415297    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415334    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415346    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415358    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415370    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415384    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415395    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415408    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.415676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.415420    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwqn9\" (UniqueName: \"kubernetes.io/projected/14ba8a64-73a1-4122-b4f0-d3b1579aaad4-kube-api-access-vwqn9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:31.848686 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.848606    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"]
Nov 02 17:52:31.849358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.849337    5042 cpu_manager.go:414] "containerMap: removing container" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy"
Nov 02 17:52:31.849496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.849360    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy"
Nov 02 17:52:31.849597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.849583    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy"
Nov 02 17:52:31.854448 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.854423    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:31.861197 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.861171    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:52:31.861436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.861227    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:52:31.861436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.861301    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:52:31.861436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.861380    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:52:31.870029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.869314    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"]
Nov 02 17:52:31.925376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.925333    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:31.925671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.925645    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqfz5\" (UniqueName: \"kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:31.925870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:31.925814    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.027015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.026975    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.027210 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.027075    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.027210 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.027205    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqfz5\" (UniqueName: \"kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.027778 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.027755    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.029681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.029658    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.036557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.036528    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqfz5\" (UniqueName: \"kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5\") pod \"maas-api-revoke-sub-b1e96f4b-62scq\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.188343 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.188310    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:32.197897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.197815    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9" event={"ID":"14ba8a64-73a1-4122-b4f0-d3b1579aaad4","Type":"ContainerDied","Data":"f899c643fcfe99d57e48362d8ae4fd2c9830867574f8d8d29f0f5b088c17792d"}
Nov 02 17:52:32.198099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.197929    5042 scope.go:122] "RemoveContainer" containerID="d353e3f69ad6b00f86a7f4beaeca6aa2c0982e9f32ef564315290cccaeeb69ad"
Nov 02 17:52:32.198099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.197958    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"
Nov 02 17:52:32.231751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.231716    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"]
Nov 02 17:52:32.239921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.239893    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-3568c5-openshift-default-5c5b74796c-nwgw9"]
Nov 02 17:52:32.507015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:32.506980    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"]
Nov 02 17:52:32.508199 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:52:32.508163    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb3bd88_f670_4767_9c08_0a38995e3679.slice/crio-a0bf198604c5ba9fd7f0d0bbdd7b698f668137c74728f3c9200af91f6767f2ef WatchSource:0}: Error finding container a0bf198604c5ba9fd7f0d0bbdd7b698f668137c74728f3c9200af91f6767f2ef: Status 404 returned error can't find the container with id a0bf198604c5ba9fd7f0d0bbdd7b698f668137c74728f3c9200af91f6767f2ef
Nov 02 17:52:33.209010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:33.208968    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq" event={"ID":"bcb3bd88-f670-4767-9c08-0a38995e3679","Type":"ContainerStarted","Data":"ae6cfc4c3e6ea667c26e38fd3338154d92b70fa20504ed9f7a44f60721d27f92"}
Nov 02 17:52:33.209221 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:33.209042    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq" event={"ID":"bcb3bd88-f670-4767-9c08-0a38995e3679","Type":"ContainerStarted","Data":"a0bf198604c5ba9fd7f0d0bbdd7b698f668137c74728f3c9200af91f6767f2ef"}
Nov 02 17:52:33.232645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:33.232593    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq" podStartSLOduration=2.232578594 podStartE2EDuration="2.232578594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:33.229113704 +0000 UTC m=+1165.808523102" watchObservedRunningTime="2026-11-02 17:52:33.232578594 +0000 UTC m=+1165.811987982"
Nov 02 17:52:33.548455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:33.548364    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" path="/var/lib/kubelet/pods/14ba8a64-73a1-4122-b4f0-d3b1579aaad4/volumes"
Nov 02 17:52:34.276553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:34.276056    5042 generic.go:356] "Generic (PLEG): container finished" podID="bcb3bd88-f670-4767-9c08-0a38995e3679" containerID="ae6cfc4c3e6ea667c26e38fd3338154d92b70fa20504ed9f7a44f60721d27f92" exitCode=0
Nov 02 17:52:34.276553 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:34.276231    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq" event={"ID":"bcb3bd88-f670-4767-9c08-0a38995e3679","Type":"ContainerDied","Data":"ae6cfc4c3e6ea667c26e38fd3338154d92b70fa20504ed9f7a44f60721d27f92"}
Nov 02 17:52:35.299895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.299294    5042 generic.go:356] "Generic (PLEG): container finished" podID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerID="eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022" exitCode=0
Nov 02 17:52:35.299895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.299734    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" event={"ID":"8ffa6cd4-155d-47ff-8d38-420db30b67cd","Type":"ContainerDied","Data":"eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"}
Nov 02 17:52:35.300525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.300242    5042 scope.go:122] "RemoveContainer" containerID="eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"
Nov 02 17:52:35.302359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.302327    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:35.302583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.302563    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:35.302706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.302693    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:35.666503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.666469    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:35.737963 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.737927    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5\" (UniqueName: \"kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5\") pod \"bcb3bd88-f670-4767-9c08-0a38995e3679\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") "
Nov 02 17:52:35.738163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.737984    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca\") pod \"bcb3bd88-f670-4767-9c08-0a38995e3679\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") "
Nov 02 17:52:35.738284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.738230    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca\") pod \"bcb3bd88-f670-4767-9c08-0a38995e3679\" (UID: \"bcb3bd88-f670-4767-9c08-0a38995e3679\") "
Nov 02 17:52:35.738437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.738409    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca" pod "bcb3bd88-f670-4767-9c08-0a38995e3679" (UID: "bcb3bd88-f670-4767-9c08-0a38995e3679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:35.739212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.739185    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:35.740477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.740446    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca" pod "bcb3bd88-f670-4767-9c08-0a38995e3679" (UID: "bcb3bd88-f670-4767-9c08-0a38995e3679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:52:35.740762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.740741    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5" pod "bcb3bd88-f670-4767-9c08-0a38995e3679" (UID: "bcb3bd88-f670-4767-9c08-0a38995e3679"). InnerVolumeSpecName "kube-api-access-wqfz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:35.840810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.840769    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb3bd88-f670-4767-9c08-0a38995e3679-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:35.840810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:35.840805    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqfz5\" (UniqueName: \"kubernetes.io/projected/bcb3bd88-f670-4767-9c08-0a38995e3679-kube-api-access-wqfz5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:36.310919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.310877    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" event={"ID":"8ffa6cd4-155d-47ff-8d38-420db30b67cd","Type":"ContainerStarted","Data":"024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7"}
Nov 02 17:52:36.312745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.312711    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq" event={"ID":"bcb3bd88-f670-4767-9c08-0a38995e3679","Type":"ContainerDied","Data":"a0bf198604c5ba9fd7f0d0bbdd7b698f668137c74728f3c9200af91f6767f2ef"}
Nov 02 17:52:36.312939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.312766    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0bf198604c5ba9fd7f0d0bbdd7b698f668137c74728f3c9200af91f6767f2ef"
Nov 02 17:52:36.312939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.312827    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"
Nov 02 17:52:36.402889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.388519    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:52:36.402889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.388662    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:36.402889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:36.388705    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:37.395868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:37.395312    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:37.395868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:37.395382    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:38.062616 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:38.062560    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"]
Nov 02 17:52:38.063200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:38.063154    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy" containerID="cri-o://ba116e2e8972a30dd1a1485552dbe39a88cbfc12a0d1d6db8ca36297bbd18429" gracePeriod=30
Nov 02 17:52:38.070651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:38.070619    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"]
Nov 02 17:52:38.070883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:38.070823    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy" containerID="cri-o://0490fcbf60bd5581eea31e50d8ba056aea58afe06ce10062e31fe9ce44c5c6e0" gracePeriod=30
Nov 02 17:52:38.391933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:38.390434    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:38.391933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:38.390496    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:39.044191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:39.043020    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:52:39.385825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:39.384958    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:39.385825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:39.385021    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:40.390874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:40.386310    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:40.390874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:40.386392    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:41.396867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:41.387286    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:41.396867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:41.387350    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:42.033486 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:42.032547    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"]
Nov 02 17:52:42.441935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:42.440225    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Nov 02 17:52:42.441935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:42.440300    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Nov 02 17:52:43.468861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.468799    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:52:43.473771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.473698    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:52:43.665907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.665824    5042 generic.go:356] "Generic (PLEG): container finished" podID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerID="0490fcbf60bd5581eea31e50d8ba056aea58afe06ce10062e31fe9ce44c5c6e0" exitCode=0
Nov 02 17:52:43.666329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.666305    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" event={"ID":"9cfe1f83-d393-4cd4-854d-07ab396dbb0e","Type":"ContainerDied","Data":"0490fcbf60bd5581eea31e50d8ba056aea58afe06ce10062e31fe9ce44c5c6e0"}
Nov 02 17:52:43.671229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.670584    5042 generic.go:356] "Generic (PLEG): container finished" podID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerID="ba116e2e8972a30dd1a1485552dbe39a88cbfc12a0d1d6db8ca36297bbd18429" exitCode=0
Nov 02 17:52:43.671229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.671011    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" event={"ID":"c235e191-4fc2-4e92-af3c-c2de4b59c9ff","Type":"ContainerDied","Data":"ba116e2e8972a30dd1a1485552dbe39a88cbfc12a0d1d6db8ca36297bbd18429"}
Nov 02 17:52:43.672607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.672585    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:52:43.674828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.674800    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:52:43.856692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856605    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.856898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856718    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.856898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856761    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.856898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856804    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.856898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856860    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856915    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.856975    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857032    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.857269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857128    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.857269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857156    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.857269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857200    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857273    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857332    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.857435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857361    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857403    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.857663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857478    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.857663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857515    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket\") pod \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\" (UID: \"9cfe1f83-d393-4cd4-854d-07ab396dbb0e\") "
Nov 02 17:52:43.857663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857547    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data\") pod \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\" (UID: \"c235e191-4fc2-4e92-af3c-c2de4b59c9ff\") "
Nov 02 17:52:43.858646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857121    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.858646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.857361    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.858646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.858301    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.859201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.859177    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.859921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.859385    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.860449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.860058    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:43.860670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.860561    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.860812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.860792    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.861186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.861164    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.861332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.861313    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:43.861732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.861686    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.861961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.861928    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:43.862986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.862960    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:43.863108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.863021    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "kube-api-access-bsg5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:43.863108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.863088    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:43.863336 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.863316    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "kube-api-access-rxh5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:43.863434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.863335    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo" pod "c235e191-4fc2-4e92-af3c-c2de4b59c9ff" (UID: "c235e191-4fc2-4e92-af3c-c2de4b59c9ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:43.863434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.863331    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy" pod "9cfe1f83-d393-4cd4-854d-07ab396dbb0e" (UID: "9cfe1f83-d393-4cd4-854d-07ab396dbb0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:43.961217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961173    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961213    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961231    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxh5t\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-kube-api-access-rxh5t\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961246    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961258    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961270    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961282    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961295    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsg5v\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-kube-api-access-bsg5v\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961308    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961319    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961331    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961342    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961355    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961367    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961379    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961400    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961412    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c235e191-4fc2-4e92-af3c-c2de4b59c9ff-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:43.961494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:43.961423    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfe1f83-d393-4cd4-854d-07ab396dbb0e-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:44.779422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.779377    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn" event={"ID":"9cfe1f83-d393-4cd4-854d-07ab396dbb0e","Type":"ContainerDied","Data":"ffc70da4594a9ba607516d0077cf3afc92305ebcf61ff259134a6ed4c7d8fae7"}
Nov 02 17:52:44.779997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.779617    5042 scope.go:122] "RemoveContainer" containerID="0490fcbf60bd5581eea31e50d8ba056aea58afe06ce10062e31fe9ce44c5c6e0"
Nov 02 17:52:44.779997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.779813    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"
Nov 02 17:52:44.804741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.803355    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl" event={"ID":"c235e191-4fc2-4e92-af3c-c2de4b59c9ff","Type":"ContainerDied","Data":"3c46fcb6d705602fb66f6d4e5c732c616a84f87d6252993b5b89d14044e93a74"}
Nov 02 17:52:44.804741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.803472    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"
Nov 02 17:52:44.816954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.816926    5042 scope.go:122] "RemoveContainer" containerID="ba116e2e8972a30dd1a1485552dbe39a88cbfc12a0d1d6db8ca36297bbd18429"
Nov 02 17:52:44.833514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.833009    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"]
Nov 02 17:52:44.838530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.838503    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-25xtn"]
Nov 02 17:52:44.856629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.856599    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"]
Nov 02 17:52:44.863446 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:44.863413    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-qsfpl"]
Nov 02 17:52:45.074931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.073892    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"]
Nov 02 17:52:45.117866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.097905    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1df0b0c-p27ql"]
Nov 02 17:52:45.349047 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.348960    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"]
Nov 02 17:52:45.349560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.349275    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5" podUID="98e1b231-b167-4dc3-8820-0126f6a8d008" containerName="payload-processing" containerID="cri-o://50e62e089568b27e182b0356fc701e552e73fba423dc1b867d481b8b8bfccbbc" gracePeriod=30
Nov 02 17:52:45.365541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.365288    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"]
Nov 02 17:52:45.369888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.365787    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn" podUID="7d0066fa-1b96-4961-a57b-b616ef3f4dab" containerName="payload-pre-processing" containerID="cri-o://bbd1b5ac8a020ca238b088fffb1849bc61a2e3d20d6a100d4afe78f4d4b734c0" gracePeriod=30
Nov 02 17:52:45.674562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.674458    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="240d54c5-72bb-477e-a175-f5fef9c365f7" path="/var/lib/kubelet/pods/240d54c5-72bb-477e-a175-f5fef9c365f7/volumes"
Nov 02 17:52:45.681590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.681546    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" path="/var/lib/kubelet/pods/9cfe1f83-d393-4cd4-854d-07ab396dbb0e/volumes"
Nov 02 17:52:45.682687 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.682663    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" path="/var/lib/kubelet/pods/c235e191-4fc2-4e92-af3c-c2de4b59c9ff/volumes"
Nov 02 17:52:45.847896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.844831    5042 generic.go:356] "Generic (PLEG): container finished" podID="98e1b231-b167-4dc3-8820-0126f6a8d008" containerID="50e62e089568b27e182b0356fc701e552e73fba423dc1b867d481b8b8bfccbbc" exitCode=0
Nov 02 17:52:45.847896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.844997    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5" event={"ID":"98e1b231-b167-4dc3-8820-0126f6a8d008","Type":"ContainerDied","Data":"50e62e089568b27e182b0356fc701e552e73fba423dc1b867d481b8b8bfccbbc"}
Nov 02 17:52:45.855442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.855029    5042 generic.go:356] "Generic (PLEG): container finished" podID="7d0066fa-1b96-4961-a57b-b616ef3f4dab" containerID="bbd1b5ac8a020ca238b088fffb1849bc61a2e3d20d6a100d4afe78f4d4b734c0" exitCode=0
Nov 02 17:52:45.855442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:45.855151    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn" event={"ID":"7d0066fa-1b96-4961-a57b-b616ef3f4dab","Type":"ContainerDied","Data":"bbd1b5ac8a020ca238b088fffb1849bc61a2e3d20d6a100d4afe78f4d4b734c0"}
Nov 02 17:52:46.159884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.159403    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:52:46.162168 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.162145    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:52:46.336451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.336357    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume\") pod \"98e1b231-b167-4dc3-8820-0126f6a8d008\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") "
Nov 02 17:52:46.336639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.336459    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume\") pod \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") "
Nov 02 17:52:46.336639 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.336574    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7\" (UniqueName: \"kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7\") pod \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\" (UID: \"7d0066fa-1b96-4961-a57b-b616ef3f4dab\") "
Nov 02 17:52:46.336761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.336642    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv\" (UniqueName: \"kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv\") pod \"98e1b231-b167-4dc3-8820-0126f6a8d008\" (UID: \"98e1b231-b167-4dc3-8820-0126f6a8d008\") "
Nov 02 17:52:46.336898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.336822    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume" pod "98e1b231-b167-4dc3-8820-0126f6a8d008" (UID: "98e1b231-b167-4dc3-8820-0126f6a8d008"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:46.337772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.337292    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e1b231-b167-4dc3-8820-0126f6a8d008-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:46.341602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.341230    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume" pod "7d0066fa-1b96-4961-a57b-b616ef3f4dab" (UID: "7d0066fa-1b96-4961-a57b-b616ef3f4dab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:46.342466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.342290    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7" pod "7d0066fa-1b96-4961-a57b-b616ef3f4dab" (UID: "7d0066fa-1b96-4961-a57b-b616ef3f4dab"). InnerVolumeSpecName "kube-api-access-b69j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:46.343337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.343103    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv" pod "98e1b231-b167-4dc3-8820-0126f6a8d008" (UID: "98e1b231-b167-4dc3-8820-0126f6a8d008"). InnerVolumeSpecName "kube-api-access-scfdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:46.438468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.438434    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0066fa-1b96-4961-a57b-b616ef3f4dab-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:46.438468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.438475    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b69j7\" (UniqueName: \"kubernetes.io/projected/7d0066fa-1b96-4961-a57b-b616ef3f4dab-kube-api-access-b69j7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:46.438675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.438488    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scfdv\" (UniqueName: \"kubernetes.io/projected/98e1b231-b167-4dc3-8820-0126f6a8d008-kube-api-access-scfdv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:46.864054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.864023    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"
Nov 02 17:52:46.864576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.864069    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5" event={"ID":"98e1b231-b167-4dc3-8820-0126f6a8d008","Type":"ContainerDied","Data":"c3e0315e247247857514713ceca54827062b54001b1eab7508edb369c8b041a4"}
Nov 02 17:52:46.864576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.864131    5042 scope.go:122] "RemoveContainer" containerID="50e62e089568b27e182b0356fc701e552e73fba423dc1b867d481b8b8bfccbbc"
Nov 02 17:52:46.866068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.866041    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"
Nov 02 17:52:46.866175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.866148    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn" event={"ID":"7d0066fa-1b96-4961-a57b-b616ef3f4dab","Type":"ContainerDied","Data":"b15abc27adbdb8c10ca93bc65f0cf2fba06fae7110c234b1eab2354204d4eab7"}
Nov 02 17:52:46.880270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.880246    5042 scope.go:122] "RemoveContainer" containerID="bbd1b5ac8a020ca238b088fffb1849bc61a2e3d20d6a100d4afe78f4d4b734c0"
Nov 02 17:52:46.954464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.954407    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"]
Nov 02 17:52:46.959428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.959398    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4c9298-f5bc5b98d-spsf5"]
Nov 02 17:52:46.967038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.963689    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"]
Nov 02 17:52:46.967230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:46.967173    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4c9298-5bb5947cf9-rllcn"]
Nov 02 17:52:47.033824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.033751    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w" podUID="03cc72d0-0825-4365-bf3f-50dfaccc32dd" containerName="payload-processing" containerID="cri-o://fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250" gracePeriod=25
Nov 02 17:52:47.419960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.419888    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:52:47.461424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.461391    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume\") pod \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") "
Nov 02 17:52:47.463162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.463120    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5\" (UniqueName: \"kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5\") pod \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\" (UID: \"03cc72d0-0825-4365-bf3f-50dfaccc32dd\") "
Nov 02 17:52:47.463713 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.463689    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume" pod "03cc72d0-0825-4365-bf3f-50dfaccc32dd" (UID: "03cc72d0-0825-4365-bf3f-50dfaccc32dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:47.464139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.464114    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03cc72d0-0825-4365-bf3f-50dfaccc32dd-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:47.466001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.465973    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5" pod "03cc72d0-0825-4365-bf3f-50dfaccc32dd" (UID: "03cc72d0-0825-4365-bf3f-50dfaccc32dd"). InnerVolumeSpecName "kube-api-access-nldg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:47.547012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.546976    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0066fa-1b96-4961-a57b-b616ef3f4dab" path="/var/lib/kubelet/pods/7d0066fa-1b96-4961-a57b-b616ef3f4dab/volumes"
Nov 02 17:52:47.547555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.547537    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e1b231-b167-4dc3-8820-0126f6a8d008" path="/var/lib/kubelet/pods/98e1b231-b167-4dc3-8820-0126f6a8d008/volumes"
Nov 02 17:52:47.566225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.566196    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldg5\" (UniqueName: \"kubernetes.io/projected/03cc72d0-0825-4365-bf3f-50dfaccc32dd-kube-api-access-nldg5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:47.876255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.876214    5042 generic.go:356] "Generic (PLEG): container finished" podID="03cc72d0-0825-4365-bf3f-50dfaccc32dd" containerID="fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250" exitCode=0
Nov 02 17:52:47.876808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.876302    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"
Nov 02 17:52:47.876808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.876301    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w" event={"ID":"03cc72d0-0825-4365-bf3f-50dfaccc32dd","Type":"ContainerDied","Data":"fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250"}
Nov 02 17:52:47.876808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.876368    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w" event={"ID":"03cc72d0-0825-4365-bf3f-50dfaccc32dd","Type":"ContainerDied","Data":"9f8acedd7ca70f7b33863e13fbaea2c944a5d53a06e3f75188186e4c24a197d6"}
Nov 02 17:52:47.876808 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.876397    5042 scope.go:122] "RemoveContainer" containerID="fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250"
Nov 02 17:52:47.890794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.890773    5042 scope.go:122] "RemoveContainer" containerID="fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250"
Nov 02 17:52:47.891191 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:47.891164    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250\": container with ID starting with fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250 not found: ID does not exist" containerID="fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250"
Nov 02 17:52:47.891260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.891202    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250"} err="failed to get container status \"fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250\": rpc error: code = NotFound desc = could not find container \"fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250\": container with ID starting with fdf2faafd0372367ccfdd89dec4929af232754b23a3d4e908d43b3119270e250 not found: ID does not exist"
Nov 02 17:52:47.920887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.901993    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"]
Nov 02 17:52:47.920887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:47.905964    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7dbdb8-586545b55c-b9l8w"]
Nov 02 17:52:48.079868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:48.078961    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"]
Nov 02 17:52:48.079868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:48.079217    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh" podUID="7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" containerName="istio-proxy" containerID="cri-o://ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817" gracePeriod=30
Nov 02 17:52:48.583812 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:48.583738    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:52:49.232555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.232530    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:52:49.330663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330584    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.330892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330672    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.330892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330741    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.330892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330771    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.330892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330812    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.330892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330872    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.331179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330998    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.331179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.331056    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.331296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.331196    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket\") pod \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\" (UID: \"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c\") "
Nov 02 17:52:49.331296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.330994    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:49.331296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.331235    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:49.331296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.331196    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:49.331600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.331531    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:49.331733 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.331557    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:49.332155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.332128    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.332267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.332162    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.332267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.332177    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.332267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.332191    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.332267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.332205    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.333298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.333270    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:49.333405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.333305    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:49.333405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.333319    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:49.333405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.333378    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs" pod "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" (UID: "7166b0a0-a0e4-4ee4-b1f1-518bc231f49c"). InnerVolumeSpecName "kube-api-access-ltwhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:49.434023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.433994    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.434023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.434020    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.434023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.434031    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltwhs\" (UniqueName: \"kubernetes.io/projected/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-kube-api-access-ltwhs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.434023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.434041    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:49.545579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.545543    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cc72d0-0825-4365-bf3f-50dfaccc32dd" path="/var/lib/kubelet/pods/03cc72d0-0825-4365-bf3f-50dfaccc32dd/volumes"
Nov 02 17:52:49.905376 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.905340    5042 generic.go:356] "Generic (PLEG): container finished" podID="7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" containerID="ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817" exitCode=0
Nov 02 17:52:49.905579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.905406    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"
Nov 02 17:52:49.905579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.905414    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh" event={"ID":"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c","Type":"ContainerDied","Data":"ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817"}
Nov 02 17:52:49.905579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.905467    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh" event={"ID":"7166b0a0-a0e4-4ee4-b1f1-518bc231f49c","Type":"ContainerDied","Data":"e34d8519a98f3b7b633cea3fb393769c804717a25bee421c87469235ecd1fe40"}
Nov 02 17:52:49.905579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.905496    5042 scope.go:122] "RemoveContainer" containerID="ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817"
Nov 02 17:52:49.916533 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.916422    5042 scope.go:122] "RemoveContainer" containerID="ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817"
Nov 02 17:52:49.917294 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:49.917261    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817\": container with ID starting with ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817 not found: ID does not exist" containerID="ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817"
Nov 02 17:52:49.917431 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.917304    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817"} err="failed to get container status \"ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817\": rpc error: code = NotFound desc = could not find container \"ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817\": container with ID starting with ebff8f5b5055e657a0689e162d4b9ef4d00f727a3d10d175183eaea108c96817 not found: ID does not exist"
Nov 02 17:52:49.929723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.929691    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"]
Nov 02 17:52:49.933205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:49.933179    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-e4fdc1-openshift-default-8476767889-d9fkh"]
Nov 02 17:52:51.556950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:51.556908    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" path="/var/lib/kubelet/pods/7166b0a0-a0e4-4ee4-b1f1-518bc231f49c/volumes"
Nov 02 17:52:53.034496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:53.034458    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"]
Nov 02 17:52:53.035170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:53.034742    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy" containerID="cri-o://265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63" gracePeriod=30
Nov 02 17:52:57.600629 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:57.600594    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"]
Nov 02 17:52:58.473821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.473789    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:52:58.569048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569014    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569122    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569150    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569232 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569171    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569320    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569443    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:58.569535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569483    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569595 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569570    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:58.569648 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569605    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569648 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569633    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569753 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569662    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket\") pod \"04a77187-c64b-4931-8aa4-4b416902f804\" (UID: \"04a77187-c64b-4931-8aa4-4b416902f804\") "
Nov 02 17:52:58.569883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.569858    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:58.570091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570037    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:58.570234 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570216    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:58.570355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570341    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.570404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570359    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.570404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570369    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.570404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570378    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04a77187-c64b-4931-8aa4-4b416902f804-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.570404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.570388    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.572095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.572067    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:58.572217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.572188    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:58.572286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.572218    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:58.572987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.572964    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk" pod "04a77187-c64b-4931-8aa4-4b416902f804" (UID: "04a77187-c64b-4931-8aa4-4b416902f804"). InnerVolumeSpecName "kube-api-access-ldqvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:58.672067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.672021    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04a77187-c64b-4931-8aa4-4b416902f804-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.672067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.672057    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04a77187-c64b-4931-8aa4-4b416902f804-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.672540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.672075    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqvk\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-kube-api-access-ldqvk\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.672540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.672093    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04a77187-c64b-4931-8aa4-4b416902f804-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:52:58.993745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.993711    5042 generic.go:356] "Generic (PLEG): container finished" podID="04a77187-c64b-4931-8aa4-4b416902f804" containerID="265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63" exitCode=0
Nov 02 17:52:58.993988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.993766    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"
Nov 02 17:52:58.993988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.993795    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" event={"ID":"04a77187-c64b-4931-8aa4-4b416902f804","Type":"ContainerDied","Data":"265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63"}
Nov 02 17:52:58.993988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.993895    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz" event={"ID":"04a77187-c64b-4931-8aa4-4b416902f804","Type":"ContainerDied","Data":"ab03578e339cf55c77b0eb1cb907635c71b1f42606afb31d903b68477a7f72c3"}
Nov 02 17:52:58.993988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:58.993921    5042 scope.go:122] "RemoveContainer" containerID="265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63"
Nov 02 17:52:59.010083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:59.010055    5042 scope.go:122] "RemoveContainer" containerID="265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63"
Nov 02 17:52:59.010454 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:52:59.010432    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63\": container with ID starting with 265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63 not found: ID does not exist" containerID="265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63"
Nov 02 17:52:59.010544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:59.010466    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63"} err="failed to get container status \"265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63\": rpc error: code = NotFound desc = could not find container \"265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63\": container with ID starting with 265eb7fe7ca1f47b24f5250f7d3541b4bf6169f0d55fb9558341d403f1fc6c63 not found: ID does not exist"
Nov 02 17:52:59.034367 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:59.034329    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"]
Nov 02 17:52:59.040050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:59.040019    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4c9298-openshift-default-6bb79c769-rdqlz"]
Nov 02 17:52:59.546201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:52:59.546170    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a77187-c64b-4931-8aa4-4b416902f804" path="/var/lib/kubelet/pods/04a77187-c64b-4931-8aa4-4b416902f804/volumes"
Nov 02 17:53:02.601348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.601303    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm" podUID="095bbff2-398e-4520-9e8a-2fe5355ed7cc" containerName="payload-processing" containerID="cri-o://a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d" gracePeriod=25
Nov 02 17:53:02.915377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.915344    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:53:02.950613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.950580    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume\") pod \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") "
Nov 02 17:53:02.950816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.950649    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm\" (UniqueName: \"kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm\") pod \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\" (UID: \"095bbff2-398e-4520-9e8a-2fe5355ed7cc\") "
Nov 02 17:53:02.951060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.951036    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume" pod "095bbff2-398e-4520-9e8a-2fe5355ed7cc" (UID: "095bbff2-398e-4520-9e8a-2fe5355ed7cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:02.951245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.951224    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/095bbff2-398e-4520-9e8a-2fe5355ed7cc-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:02.954937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:02.953526    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm" pod "095bbff2-398e-4520-9e8a-2fe5355ed7cc" (UID: "095bbff2-398e-4520-9e8a-2fe5355ed7cc"). InnerVolumeSpecName "kube-api-access-74ccm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:03.045355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.045317    5042 generic.go:356] "Generic (PLEG): container finished" podID="095bbff2-398e-4520-9e8a-2fe5355ed7cc" containerID="a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d" exitCode=0
Nov 02 17:53:03.045511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.045369    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"
Nov 02 17:53:03.045511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.045377    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm" event={"ID":"095bbff2-398e-4520-9e8a-2fe5355ed7cc","Type":"ContainerDied","Data":"a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d"}
Nov 02 17:53:03.045511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.045446    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm" event={"ID":"095bbff2-398e-4520-9e8a-2fe5355ed7cc","Type":"ContainerDied","Data":"4c663d272fb62f10b3c174c0cebadd744220a27aed4a3242db14d0a97e8ca052"}
Nov 02 17:53:03.045511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.045467    5042 scope.go:122] "RemoveContainer" containerID="a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d"
Nov 02 17:53:03.052364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.052340    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ccm\" (UniqueName: \"kubernetes.io/projected/095bbff2-398e-4520-9e8a-2fe5355ed7cc-kube-api-access-74ccm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:03.056419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.056399    5042 scope.go:122] "RemoveContainer" containerID="a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d"
Nov 02 17:53:03.056719 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:53:03.056702    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d\": container with ID starting with a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d not found: ID does not exist" containerID="a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d"
Nov 02 17:53:03.056770 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.056730    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d"} err="failed to get container status \"a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d\": rpc error: code = NotFound desc = could not find container \"a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d\": container with ID starting with a1057e06cac4339183994d5c644d3d9d24ffee3557e3a4bb29fd1840ef72e87d not found: ID does not exist"
Nov 02 17:53:03.082389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.082348    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"]
Nov 02 17:53:03.092471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.092442    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4a5f09-5f5d6ffbff-vxtkm"]
Nov 02 17:53:03.544478 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:03.544441    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095bbff2-398e-4520-9e8a-2fe5355ed7cc" path="/var/lib/kubelet/pods/095bbff2-398e-4520-9e8a-2fe5355ed7cc/volumes"
Nov 02 17:53:04.071199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071163    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"]
Nov 02 17:53:04.071735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071719    5042 cpu_manager.go:414] "containerMap: removing container" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy"
Nov 02 17:53:04.071735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071734    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071752    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071757    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071768    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7d0066fa-1b96-4961-a57b-b616ef3f4dab" containerName="payload-pre-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071773    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0066fa-1b96-4961-a57b-b616ef3f4dab" containerName="payload-pre-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071794    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071799    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071810    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071815    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071823    5042 cpu_manager.go:414] "containerMap: removing container" podUID="98e1b231-b167-4dc3-8820-0126f6a8d008" containerName="payload-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071828    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e1b231-b167-4dc3-8820-0126f6a8d008" containerName="payload-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071853    5042 cpu_manager.go:414] "containerMap: removing container" podUID="03cc72d0-0825-4365-bf3f-50dfaccc32dd" containerName="payload-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071858    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cc72d0-0825-4365-bf3f-50dfaccc32dd" containerName="payload-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071868    5042 cpu_manager.go:414] "containerMap: removing container" podUID="095bbff2-398e-4520-9e8a-2fe5355ed7cc" containerName="payload-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071872    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095bbff2-398e-4520-9e8a-2fe5355ed7cc" containerName="payload-processing"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071880    5042 cpu_manager.go:414] "containerMap: removing container" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071884    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071894    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bcb3bd88-f670-4767-9c08-0a38995e3679" containerName="revoke-keys"
Nov 02 17:53:04.071931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071898    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb3bd88-f670-4767-9c08-0a38995e3679" containerName="revoke-keys"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.071993    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb3bd88-f670-4767-9c08-0a38995e3679" containerName="revoke-keys"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072003    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a77187-c64b-4931-8aa4-4b416902f804" containerName="istio-proxy"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072010    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba8a64-73a1-4122-b4f0-d3b1579aaad4" containerName="istio-proxy"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072017    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7166b0a0-a0e4-4ee4-b1f1-518bc231f49c" containerName="istio-proxy"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072023    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0066fa-1b96-4961-a57b-b616ef3f4dab" containerName="payload-pre-processing"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072031    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cfe1f83-d393-4cd4-854d-07ab396dbb0e" containerName="istio-proxy"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072038    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c235e191-4fc2-4e92-af3c-c2de4b59c9ff" containerName="istio-proxy"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072043    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e1b231-b167-4dc3-8820-0126f6a8d008" containerName="payload-processing"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072050    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cc72d0-0825-4365-bf3f-50dfaccc32dd" containerName="payload-processing"
Nov 02 17:53:04.072855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.072055    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="095bbff2-398e-4520-9e8a-2fe5355ed7cc" containerName="payload-processing"
Nov 02 17:53:04.076909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.076880    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.118746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.118694    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"]
Nov 02 17:53:04.165870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.146349    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eda8e2\""
Nov 02 17:53:04.165870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.165228    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eda8e2-dockercfg-ksz57\""
Nov 02 17:53:04.250866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.250757    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5px7\" (UniqueName: \"kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.251048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.250945    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.353918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.353604    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.353918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.353880    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5px7\" (UniqueName: \"kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.358962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.355255    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.368241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.368209    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5px7\" (UniqueName: \"kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7\") pod \"payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.408801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.408763    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:04.491621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.491592    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"]
Nov 02 17:53:04.523707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.522739    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.524773 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.524750    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"]
Nov 02 17:53:04.528281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.528176    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-6f78de-openshift-default-dockercfg-pjh95\""
Nov 02 17:53:04.557760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.557717    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.557958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.557778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qm2\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.557958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.557827    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.557958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.557873    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.557958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.557937    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.558174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.557987    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.558174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.558068    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.559298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.558377    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.559298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.558418    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660147    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660218    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660307    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660325    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660358    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660378    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qm2\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660419    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660446    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660756 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660518    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660756 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660687    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.660833 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660798    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.661017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.660997    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.661093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.661043    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.661093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.661067    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.662986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.662967    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.663248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.663231    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.669001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.668981    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.669144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.669124    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qm2\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2\") pod \"e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") " pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:04.734151 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:53:04.734115    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8376352d_916c_4ebe_b4e0_01317cd52091.slice/crio-b6665c8f7a7a39fb9ba44783322c7ce558476018fa8e7d1f702fe6010cdad611 WatchSource:0}: Error finding container b6665c8f7a7a39fb9ba44783322c7ce558476018fa8e7d1f702fe6010cdad611: Status 404 returned error can't find the container with id b6665c8f7a7a39fb9ba44783322c7ce558476018fa8e7d1f702fe6010cdad611
Nov 02 17:53:04.736708 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.736599    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"]
Nov 02 17:53:04.875878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:04.874614    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:05.066906 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.066827    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd" event={"ID":"8376352d-916c-4ebe-b4e0-01317cd52091","Type":"ContainerStarted","Data":"c05021a50d16ee36692867e7b3680f86f2828615d3a1a323c2c135f95606c8d0"}
Nov 02 17:53:05.067087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.066926    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd" event={"ID":"8376352d-916c-4ebe-b4e0-01317cd52091","Type":"ContainerStarted","Data":"b6665c8f7a7a39fb9ba44783322c7ce558476018fa8e7d1f702fe6010cdad611"}
Nov 02 17:53:05.067827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.067804    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:05.097871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.097789    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd" podStartSLOduration=1.097768474 podStartE2EDuration="1.097768474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:05.092567198 +0000 UTC m=+1197.671976585" watchObservedRunningTime="2026-11-02 17:53:05.097768474 +0000 UTC m=+1197.677177862"
Nov 02 17:53:05.248186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.238536    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"]
Nov 02 17:53:05.262941 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:53:05.259992    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c8d6f7_64d7_47d4_a5dc_7ccd60650eb6.slice/crio-a5c6690da7053f66e6c24a5bedbb99ec66c0ada075eacff31eb72e1b36997028 WatchSource:0}: Error finding container a5c6690da7053f66e6c24a5bedbb99ec66c0ada075eacff31eb72e1b36997028: Status 404 returned error can't find the container with id a5c6690da7053f66e6c24a5bedbb99ec66c0ada075eacff31eb72e1b36997028
Nov 02 17:53:05.264976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.264913    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:05.265161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.265010    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:05.265161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:05.265055    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:06.080309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:06.080269    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" event={"ID":"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6","Type":"ContainerStarted","Data":"846543a8927fd97c3dea5d16376205e81340f4c02cc45eb3d8233061700dceae"}
Nov 02 17:53:06.080547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:06.080342    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" event={"ID":"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6","Type":"ContainerStarted","Data":"a5c6690da7053f66e6c24a5bedbb99ec66c0ada075eacff31eb72e1b36997028"}
Nov 02 17:53:06.140084 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:06.140020    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podStartSLOduration=2.140000605 podStartE2EDuration="2.140000605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:06.137064033 +0000 UTC m=+1198.716473445" watchObservedRunningTime="2026-11-02 17:53:06.140000605 +0000 UTC m=+1198.719409992"
Nov 02 17:53:06.892466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:06.892425    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:06.914238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:06.914004    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:07.108430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:07.108393    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:07.112902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:07.111244    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:53:08.102864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:08.100202    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:53:08.106863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:08.103706    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:53:08.248379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:08.248348    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:53:08.251327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:08.251306    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:53:09.769059 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:09.769026    5042 scope.go:122] "RemoveContainer" containerID="c3a9ff20473bc975bcc38993a6657164f52b980160e3111e66ac3b4b5d366992"
Nov 02 17:53:09.806324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:09.806297    5042 scope.go:122] "RemoveContainer" containerID="fb933a8799ee2a87f9e7662908b77a3f4c85485bc68c54cac94b0c7744420b73"
Nov 02 17:53:09.816501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:09.816473    5042 scope.go:122] "RemoveContainer" containerID="8c51716c83605a78bcc885620b868a9154078271825ddcd7d5339661dfdea6b7"
Nov 02 17:53:09.826895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:09.826866    5042 scope.go:122] "RemoveContainer" containerID="eff2843d215b8612ed7a12d758cd75f9a025f45e8accabd2adff27251e15388c"
Nov 02 17:53:10.280229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.270661    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"]
Nov 02 17:53:10.280229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.270929    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj" podUID="0869f30c-b58a-40ee-9108-d4c504934fd7" containerName="payload-processing" containerID="cri-o://bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16" gracePeriod=30
Nov 02 17:53:10.749196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.749173    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:53:10.761679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.761651    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume\") pod \"0869f30c-b58a-40ee-9108-d4c504934fd7\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") "
Nov 02 17:53:10.761876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.761766    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65\" (UniqueName: \"kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65\") pod \"0869f30c-b58a-40ee-9108-d4c504934fd7\" (UID: \"0869f30c-b58a-40ee-9108-d4c504934fd7\") "
Nov 02 17:53:10.762061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.762035    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume" pod "0869f30c-b58a-40ee-9108-d4c504934fd7" (UID: "0869f30c-b58a-40ee-9108-d4c504934fd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:10.762383 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.762362    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0869f30c-b58a-40ee-9108-d4c504934fd7-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:10.764070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.764047    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65" pod "0869f30c-b58a-40ee-9108-d4c504934fd7" (UID: "0869f30c-b58a-40ee-9108-d4c504934fd7"). InnerVolumeSpecName "kube-api-access-clx65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:10.868958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:10.868362    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clx65\" (UniqueName: \"kubernetes.io/projected/0869f30c-b58a-40ee-9108-d4c504934fd7-kube-api-access-clx65\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:11.166489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.166387    5042 generic.go:356] "Generic (PLEG): container finished" podID="0869f30c-b58a-40ee-9108-d4c504934fd7" containerID="bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16" exitCode=0
Nov 02 17:53:11.166489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.166445    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"
Nov 02 17:53:11.166489 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.166469    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj" event={"ID":"0869f30c-b58a-40ee-9108-d4c504934fd7","Type":"ContainerDied","Data":"bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16"}
Nov 02 17:53:11.166729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.166523    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj" event={"ID":"0869f30c-b58a-40ee-9108-d4c504934fd7","Type":"ContainerDied","Data":"f69a070100ace9ebb49275844008e6c735abec65336e8c68030e2cf5472a1fc0"}
Nov 02 17:53:11.166729 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.166545    5042 scope.go:122] "RemoveContainer" containerID="bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16"
Nov 02 17:53:11.204955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.204936    5042 scope.go:122] "RemoveContainer" containerID="bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16"
Nov 02 17:53:11.205339 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:53:11.205316    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16\": container with ID starting with bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16 not found: ID does not exist" containerID="bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16"
Nov 02 17:53:11.205428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.205350    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16"} err="failed to get container status \"bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16\": rpc error: code = NotFound desc = could not find container \"bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16\": container with ID starting with bfc257f3d42f82fdd208ff93acde567652166f23b4a65a6f3dcbeb9080891f16 not found: ID does not exist"
Nov 02 17:53:11.228032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.228002    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"]
Nov 02 17:53:11.231280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.231254    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8f7957-74f686f95b-jnvkj"]
Nov 02 17:53:11.558759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:11.558722    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0869f30c-b58a-40ee-9108-d4c504934fd7" path="/var/lib/kubelet/pods/0869f30c-b58a-40ee-9108-d4c504934fd7/volumes"
Nov 02 17:53:12.531434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:12.531401    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"]
Nov 02 17:53:12.531992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:12.531615    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" containerID="cri-o://024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7" gracePeriod=30
Nov 02 17:53:13.474105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:13.474066    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:53:13.474287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:13.474148    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:53:17.108620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.108587    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:17.213881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.211035    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:53:17.213881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.211109    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:17.914927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.910750    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:53:17.987322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987290    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987339    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987383    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987404    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987441    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987505    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987558    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987593    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987685    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket\") pod \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\" (UID: \"8ffa6cd4-155d-47ff-8d38-420db30b67cd\") "
Nov 02 17:53:17.987830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.987749    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:17.988074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.988014    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:17.988185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.988162    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:17.988542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.988515    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:17.988646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.988545    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:17.988646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.988560    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:17.988646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.988592    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:17.991017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.990993    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:17.991277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.991255    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "kube-api-access-8kcpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:17.991425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.991406    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:17.994003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.993961    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:17.994198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:17.994178    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket" pod "8ffa6cd4-155d-47ff-8d38-420db30b67cd" (UID: "8ffa6cd4-155d-47ff-8d38-420db30b67cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:18.090422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.090381    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:18.090422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.090427    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcpg\" (UniqueName: \"kubernetes.io/projected/8ffa6cd4-155d-47ff-8d38-420db30b67cd-kube-api-access-8kcpg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:18.090637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.090442    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:18.090637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.090456    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:18.090637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.090470    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ffa6cd4-155d-47ff-8d38-420db30b67cd-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:18.090637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.090483    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ffa6cd4-155d-47ff-8d38-420db30b67cd-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:18.269335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.268394    5042 generic.go:356] "Generic (PLEG): container finished" podID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerID="024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7" exitCode=0
Nov 02 17:53:18.269335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.268516    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" event={"ID":"8ffa6cd4-155d-47ff-8d38-420db30b67cd","Type":"ContainerDied","Data":"024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7"}
Nov 02 17:53:18.269335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.268670    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc" event={"ID":"8ffa6cd4-155d-47ff-8d38-420db30b67cd","Type":"ContainerDied","Data":"45ddd9e3f63d5804d9d7ab974a6884612f84ec4cd6947c6d90cebed9da28efb0"}
Nov 02 17:53:18.269335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.268715    5042 scope.go:122] "RemoveContainer" containerID="024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7"
Nov 02 17:53:18.269335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.268912    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"
Nov 02 17:53:18.279937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.279903    5042 generic.go:356] "Generic (PLEG): container finished" podID="41c8f333-4d05-40fa-a759-543d212b79e2" containerID="fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1" exitCode=0
Nov 02 17:53:18.280172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.280149    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" event={"ID":"41c8f333-4d05-40fa-a759-543d212b79e2","Type":"ContainerDied","Data":"fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"}
Nov 02 17:53:18.280679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.280657    5042 scope.go:122] "RemoveContainer" containerID="fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"
Nov 02 17:53:18.283670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.283637    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:18.283765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.283731    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:18.283858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.283785    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:18.292704 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.292685    5042 scope.go:122] "RemoveContainer" containerID="eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"
Nov 02 17:53:18.308129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.308103    5042 scope.go:122] "RemoveContainer" containerID="024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7"
Nov 02 17:53:18.308893 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:53:18.308823    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7\": container with ID starting with 024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7 not found: ID does not exist" containerID="024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7"
Nov 02 17:53:18.309015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.308903    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7"} err="failed to get container status \"024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7\": rpc error: code = NotFound desc = could not find container \"024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7\": container with ID starting with 024ef25ea8c95e7ce0af44cf750829c201cc04df61f5f9df74b8fbb040a3f5d7 not found: ID does not exist"
Nov 02 17:53:18.309015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.308925    5042 scope.go:122] "RemoveContainer" containerID="eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"
Nov 02 17:53:18.309296 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:53:18.309274    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022\": container with ID starting with eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022 not found: ID does not exist" containerID="eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"
Nov 02 17:53:18.309414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.309302    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022"} err="failed to get container status \"eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022\": rpc error: code = NotFound desc = could not find container \"eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022\": container with ID starting with eee7dd0ea9e928a026ddf530ac5673738ef75e0889976991a0be189c1d1c1022 not found: ID does not exist"
Nov 02 17:53:18.331760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.331722    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"]
Nov 02 17:53:18.340160 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:18.340128    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-009a5f-openshift-default-c547f59b5-tncxc"]
Nov 02 17:53:18.571319 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:53:18.571226    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:53:19.333203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:19.333165    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" event={"ID":"41c8f333-4d05-40fa-a759-543d212b79e2","Type":"ContainerStarted","Data":"b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580"}
Nov 02 17:53:19.544855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:19.544801    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" path="/var/lib/kubelet/pods/8ffa6cd4-155d-47ff-8d38-420db30b67cd/volumes"
Nov 02 17:53:19.653355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:19.653170    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:53:19.653672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:19.653644    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 17:53:19.653811 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:19.653703    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 17:53:20.663870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.659269    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 17:53:20.663870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.659332    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 17:53:20.962275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.962231    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"]
Nov 02 17:53:20.963649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.963617    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy"
Nov 02 17:53:20.963649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.963647    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy"
Nov 02 17:53:20.963869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.963665    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy"
Nov 02 17:53:20.963869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.963679    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy"
Nov 02 17:53:20.963869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.963711    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0869f30c-b58a-40ee-9108-d4c504934fd7" containerName="payload-processing"
Nov 02 17:53:20.963869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.963719    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0869f30c-b58a-40ee-9108-d4c504934fd7" containerName="payload-processing"
Nov 02 17:53:20.964061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.964043    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0869f30c-b58a-40ee-9108-d4c504934fd7" containerName="payload-processing"
Nov 02 17:53:20.964110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.964090    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy"
Nov 02 17:53:20.964110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.964101    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ffa6cd4-155d-47ff-8d38-420db30b67cd" containerName="istio-proxy"
Nov 02 17:53:20.970137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.970103    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:20.984866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.984010    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:53:20.984866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:20.984115    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:53:21.027821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.022032    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"]
Nov 02 17:53:21.042707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.042671    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:21.042916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.042793    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:53:21.095861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.094887    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4qz\" (UniqueName: \"kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.095861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.094969    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.095861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.095004    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.201953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.201911    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4qz\" (UniqueName: \"kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.202611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.202584    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.203613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.203592    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.204400 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.203532    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.211309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.211286    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.211995 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.211970    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4qz\" (UniqueName: \"kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz\") pod \"maas-api-revoke-sub-407aa3fa-8gwsc\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.321191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.321100    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:21.656357 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.656258    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 17:53:21.656357 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.656319    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 17:53:21.821049 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:21.819761    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"]
Nov 02 17:53:22.331292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.331200    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"]
Nov 02 17:53:22.335563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.335540    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.348752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.348714    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4e6649-openshift-default-dockercfg-9rfvj\""
Nov 02 17:53:22.354365 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.354338    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"]
Nov 02 17:53:22.484280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.484241    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.554981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555067    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555122    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555243    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555276    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555306    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555446    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.555539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.555519    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mltqt\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.708931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.679701    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mltqt\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.712867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.709266    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.723608    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.723693    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.723743    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.723939    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.723978    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.724011    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.724168    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.724671    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.722712    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.722748    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.725049    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.726345    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.728754    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.739858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.739567    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.740216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.740157    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.744934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.742367    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.752432 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.752373    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mltqt\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.756303 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.754003    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc" event={"ID":"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22","Type":"ContainerStarted","Data":"22b85617e831fe3c4648a8b6dd81bb7cc79d11c0207f54735c6bc10ac13b51ad"}
Nov 02 17:53:22.756303 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.754073    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc" event={"ID":"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22","Type":"ContainerStarted","Data":"dc1ef68c02d16471c32285ebddbdabd2c339656ae462276b4b1bef8c2c894bfe"}
Nov 02 17:53:22.756303 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.755472    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token\") pod \"e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") " pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:22.954231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:22.954184    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:23.347543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.347472    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc" podStartSLOduration=3.34745156 podStartE2EDuration="3.34745156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:22.798150085 +0000 UTC m=+1215.377559472" watchObservedRunningTime="2026-11-02 17:53:23.34745156 +0000 UTC m=+1215.926860950"
Nov 02 17:53:23.350324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.350284    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"]
Nov 02 17:53:23.429177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.422565    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:23.429532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.429493    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:23.429714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.429693    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:23.653594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.653553    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 17:53:23.653775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.653608    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 17:53:23.765200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.765110    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" event={"ID":"24c61ce6-6b5f-4c3c-a51f-506b49ba072c","Type":"ContainerStarted","Data":"2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"}
Nov 02 17:53:23.765200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.765179    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" event={"ID":"24c61ce6-6b5f-4c3c-a51f-506b49ba072c","Type":"ContainerStarted","Data":"bdb60edd0c0e1e5f3a968e6a7f924253c9474cdfdda8ba8fb6b92806b861bf86"}
Nov 02 17:53:23.770041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.770005    5042 generic.go:356] "Generic (PLEG): container finished" podID="1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" containerID="22b85617e831fe3c4648a8b6dd81bb7cc79d11c0207f54735c6bc10ac13b51ad" exitCode=0
Nov 02 17:53:23.770219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.770107    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc" event={"ID":"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22","Type":"ContainerDied","Data":"22b85617e831fe3c4648a8b6dd81bb7cc79d11c0207f54735c6bc10ac13b51ad"}
Nov 02 17:53:23.809863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.808095    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podStartSLOduration=1.808074247 podStartE2EDuration="1.808074247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:23.804181642 +0000 UTC m=+1216.383591029" watchObservedRunningTime="2026-11-02 17:53:23.808074247 +0000 UTC m=+1216.387483633"
Nov 02 17:53:23.962871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:23.957951    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:24.653683 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:24.653643    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body=
Nov 02 17:53:24.653916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:24.653716    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused"
Nov 02 17:53:24.964508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:24.964440    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:24.993801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:24.993764    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:25.112450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.111391    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca\") pod \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") "
Nov 02 17:53:25.112450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.111481    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz\" (UniqueName: \"kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz\") pod \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") "
Nov 02 17:53:25.112450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.111609    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca\") pod \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\" (UID: \"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22\") "
Nov 02 17:53:25.113217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.113189    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca" pod "1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" (UID: "1cfc351c-8eeb-4cb4-8ae5-88babdc17a22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:25.114767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.114736    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca" pod "1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" (UID: "1cfc351c-8eeb-4cb4-8ae5-88babdc17a22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:25.115320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.115286    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz" pod "1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" (UID: "1cfc351c-8eeb-4cb4-8ae5-88babdc17a22"). InnerVolumeSpecName "kube-api-access-wc4qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:25.214340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.214250    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:25.214340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.214290    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4qz\" (UniqueName: \"kubernetes.io/projected/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-kube-api-access-wc4qz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:25.214340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.214305    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:25.657669 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.657635    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:53:25.657964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.657939    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:53:25.658746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.658726    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:53:25.811001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.810905    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"
Nov 02 17:53:25.811001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.810906    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc" event={"ID":"1cfc351c-8eeb-4cb4-8ae5-88babdc17a22","Type":"ContainerDied","Data":"dc1ef68c02d16471c32285ebddbdabd2c339656ae462276b4b1bef8c2c894bfe"}
Nov 02 17:53:25.811001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.810974    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc1ef68c02d16471c32285ebddbdabd2c339656ae462276b4b1bef8c2c894bfe"
Nov 02 17:53:25.811987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.811965    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:25.813474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:25.813456    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:53:26.050696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:26.050659    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"]
Nov 02 17:53:26.064046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:26.063970    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690925a8-xjdzj"]
Nov 02 17:53:27.546141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:27.546109    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976d7ec1-033a-48b7-b75a-8459e7a94e5c" path="/var/lib/kubelet/pods/976d7ec1-033a-48b7-b75a-8459e7a94e5c/volumes"
Nov 02 17:53:29.607737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.607704    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"]
Nov 02 17:53:29.609492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.609467    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" containerName="revoke-keys"
Nov 02 17:53:29.609674 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.609659    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" containerName="revoke-keys"
Nov 02 17:53:29.609965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.609948    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" containerName="revoke-keys"
Nov 02 17:53:29.615286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.615250    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"]
Nov 02 17:53:29.615440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.615307    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.625484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.625215    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:29.628074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.627219    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:53:29.628074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.627300    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:53:29.628074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.627380    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:53:29.798034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.797997    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gs2c\" (UniqueName: \"kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.798267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.798251    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.798333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.798280    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.899709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.899603    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.899709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.899665    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.899981 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.899774    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gs2c\" (UniqueName: \"kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.900464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.900440    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.902414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.902391    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.915366 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.915333    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gs2c\" (UniqueName: \"kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c\") pod \"maas-api-revoke-sub-19d70f47-kknrm\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:29.963927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:29.950528    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:30.286964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:30.286923    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"]
Nov 02 17:53:30.299214 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:53:30.299163    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a40814f_638c_4aa5_9ecb_84dbf1a68acf.slice/crio-22c24e9a5e8d2361feab8cd736ba7b5f8a960969f2cbc25f4b9f6176730d9494 WatchSource:0}: Error finding container 22c24e9a5e8d2361feab8cd736ba7b5f8a960969f2cbc25f4b9f6176730d9494: Status 404 returned error can't find the container with id 22c24e9a5e8d2361feab8cd736ba7b5f8a960969f2cbc25f4b9f6176730d9494
Nov 02 17:53:30.880385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:30.880336    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm" event={"ID":"2a40814f-638c-4aa5-9ecb-84dbf1a68acf","Type":"ContainerStarted","Data":"af629ae66792290d9664bd748cf7e6afbb866c326e66de4a043772bb547da9ee"}
Nov 02 17:53:30.880921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:30.880412    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm" event={"ID":"2a40814f-638c-4aa5-9ecb-84dbf1a68acf","Type":"ContainerStarted","Data":"22c24e9a5e8d2361feab8cd736ba7b5f8a960969f2cbc25f4b9f6176730d9494"}
Nov 02 17:53:30.902557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:30.902509    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm" podStartSLOduration=1.902495297 podStartE2EDuration="1.902495297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:30.900398956 +0000 UTC m=+1223.479808343" watchObservedRunningTime="2026-11-02 17:53:30.902495297 +0000 UTC m=+1223.481904682"
Nov 02 17:53:31.890073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:31.889975    5042 generic.go:356] "Generic (PLEG): container finished" podID="2a40814f-638c-4aa5-9ecb-84dbf1a68acf" containerID="af629ae66792290d9664bd748cf7e6afbb866c326e66de4a043772bb547da9ee" exitCode=0
Nov 02 17:53:31.890073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:31.890040    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm" event={"ID":"2a40814f-638c-4aa5-9ecb-84dbf1a68acf","Type":"ContainerDied","Data":"af629ae66792290d9664bd748cf7e6afbb866c326e66de4a043772bb547da9ee"}
Nov 02 17:53:33.120715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.120689    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:33.185576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.185473    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca\") pod \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") "
Nov 02 17:53:33.185757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.185671    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c\" (UniqueName: \"kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c\") pod \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") "
Nov 02 17:53:33.185757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.185737    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca\") pod \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\" (UID: \"2a40814f-638c-4aa5-9ecb-84dbf1a68acf\") "
Nov 02 17:53:33.185971 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.185948    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca" pod "2a40814f-638c-4aa5-9ecb-84dbf1a68acf" (UID: "2a40814f-638c-4aa5-9ecb-84dbf1a68acf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:33.186888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.186859    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:33.189326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.189285    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca" pod "2a40814f-638c-4aa5-9ecb-84dbf1a68acf" (UID: "2a40814f-638c-4aa5-9ecb-84dbf1a68acf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:33.189451 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.189335    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c" pod "2a40814f-638c-4aa5-9ecb-84dbf1a68acf" (UID: "2a40814f-638c-4aa5-9ecb-84dbf1a68acf"). InnerVolumeSpecName "kube-api-access-7gs2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:33.288931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.288888    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gs2c\" (UniqueName: \"kubernetes.io/projected/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-kube-api-access-7gs2c\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:33.288931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.288929    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a40814f-638c-4aa5-9ecb-84dbf1a68acf-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:33.925150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.925064    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"
Nov 02 17:53:33.925150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.925092    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm" event={"ID":"2a40814f-638c-4aa5-9ecb-84dbf1a68acf","Type":"ContainerDied","Data":"22c24e9a5e8d2361feab8cd736ba7b5f8a960969f2cbc25f4b9f6176730d9494"}
Nov 02 17:53:33.925150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:33.925147    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22c24e9a5e8d2361feab8cd736ba7b5f8a960969f2cbc25f4b9f6176730d9494"
Nov 02 17:53:34.184925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.184821    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"]
Nov 02 17:53:34.185659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.185636    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2a40814f-638c-4aa5-9ecb-84dbf1a68acf" containerName="revoke-keys"
Nov 02 17:53:34.185767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.185660    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a40814f-638c-4aa5-9ecb-84dbf1a68acf" containerName="revoke-keys"
Nov 02 17:53:34.185898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.185873    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a40814f-638c-4aa5-9ecb-84dbf1a68acf" containerName="revoke-keys"
Nov 02 17:53:34.189529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.189506    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.200475 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.200448    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.200615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.200594    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.200696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.200648    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgxw2\" (UniqueName: \"kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.201067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.201047    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:53:34.202138 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.202102    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:34.202233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.202223    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:53:34.202332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.202319    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:53:34.205405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.205380    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"]
Nov 02 17:53:34.311296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.307643    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgxw2\" (UniqueName: \"kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.311296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.307829    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.311296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.307973    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.311296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.309084    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.314126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.312824    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.331073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.331007    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgxw2\" (UniqueName: \"kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2\") pod \"maas-api-revoke-sub-c67fa0b0-bh6jg\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.504732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.504698    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:34.803518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.803491    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"]
Nov 02 17:53:34.805330 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:53:34.805294    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46c239f_2c0d_40d7_a5ff_25def0b5b3fd.slice/crio-228e3a67aac384659556b911e87fd105be3a95e6c431ff3c984e889dc9269bb6 WatchSource:0}: Error finding container 228e3a67aac384659556b911e87fd105be3a95e6c431ff3c984e889dc9269bb6: Status 404 returned error can't find the container with id 228e3a67aac384659556b911e87fd105be3a95e6c431ff3c984e889dc9269bb6
Nov 02 17:53:34.933512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:34.933471    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg" event={"ID":"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd","Type":"ContainerStarted","Data":"228e3a67aac384659556b911e87fd105be3a95e6c431ff3c984e889dc9269bb6"}
Nov 02 17:53:35.960116 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:35.960068    5042 generic.go:356] "Generic (PLEG): container finished" podID="b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" containerID="9164b27b9fe16d7eb869989ada8a4ac509567c80c785cfa8a88dda1a718a9b9a" exitCode=0
Nov 02 17:53:35.960647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:35.960225    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg" event={"ID":"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd","Type":"ContainerDied","Data":"9164b27b9fe16d7eb869989ada8a4ac509567c80c785cfa8a88dda1a718a9b9a"}
Nov 02 17:53:37.135701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.135675    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:37.273250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.273144    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca\") pod \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") "
Nov 02 17:53:37.273445 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.273280    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca\") pod \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") "
Nov 02 17:53:37.273445 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.273322    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2\" (UniqueName: \"kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2\") pod \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\" (UID: \"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd\") "
Nov 02 17:53:37.273712 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.273683    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca" pod "b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" (UID: "b46c239f-2c0d-40d7-a5ff-25def0b5b3fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:37.274048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.274031    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:37.276186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.276148    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca" pod "b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" (UID: "b46c239f-2c0d-40d7-a5ff-25def0b5b3fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:37.276186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.276165    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2" pod "b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" (UID: "b46c239f-2c0d-40d7-a5ff-25def0b5b3fd"). InnerVolumeSpecName "kube-api-access-bgxw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:37.375019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.374987    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgxw2\" (UniqueName: \"kubernetes.io/projected/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-kube-api-access-bgxw2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:37.375019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.375017    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:37.975709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.975676    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"
Nov 02 17:53:37.975709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.975702    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg" event={"ID":"b46c239f-2c0d-40d7-a5ff-25def0b5b3fd","Type":"ContainerDied","Data":"228e3a67aac384659556b911e87fd105be3a95e6c431ff3c984e889dc9269bb6"}
Nov 02 17:53:37.976006 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:37.975748    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228e3a67aac384659556b911e87fd105be3a95e6c431ff3c984e889dc9269bb6"
Nov 02 17:53:42.215477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.215440    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"]
Nov 02 17:53:42.216437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.216416    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" containerName="revoke-keys"
Nov 02 17:53:42.216512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.216441    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" containerName="revoke-keys"
Nov 02 17:53:42.216615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.216604    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" containerName="revoke-keys"
Nov 02 17:53:42.221028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.221009    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.228967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.228908    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:53:42.236881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.230926    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:53:42.236881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.231000    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:53:42.236881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.231557    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:42.236881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.233387    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"]
Nov 02 17:53:42.359703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.359665    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52959\" (UniqueName: \"kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.359907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.359828    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.360035 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.360011    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.462463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.462152    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.464193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.462829    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.464193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.463025    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52959\" (UniqueName: \"kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.464193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.464177    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.471587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.468710    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.477354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.477318    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52959\" (UniqueName: \"kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959\") pod \"maas-api-revoke-sub-7565068a-4tbm6\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.545089 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.545060    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:42.845525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:42.845498    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"]
Nov 02 17:53:42.846661 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:53:42.846632    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6be90a_cf4c_4ec4_8190_03028b6f0fea.slice/crio-9b1f9e24c1572776bfb200f873742a10452cb3139487fc753b4f21dd962e1718 WatchSource:0}: Error finding container 9b1f9e24c1572776bfb200f873742a10452cb3139487fc753b4f21dd962e1718: Status 404 returned error can't find the container with id 9b1f9e24c1572776bfb200f873742a10452cb3139487fc753b4f21dd962e1718
Nov 02 17:53:43.032077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:43.032038    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6" event={"ID":"1f6be90a-cf4c-4ec4-8190-03028b6f0fea","Type":"ContainerStarted","Data":"9b1f9e24c1572776bfb200f873742a10452cb3139487fc753b4f21dd962e1718"}
Nov 02 17:53:44.068499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.068465    5042 generic.go:356] "Generic (PLEG): container finished" podID="1f6be90a-cf4c-4ec4-8190-03028b6f0fea" containerID="e5816443496710226223ae940cd1f39a9acbe6d8494a774776c9f882036b753e" exitCode=0
Nov 02 17:53:44.068997 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.068571    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6" event={"ID":"1f6be90a-cf4c-4ec4-8190-03028b6f0fea","Type":"ContainerDied","Data":"e5816443496710226223ae940cd1f39a9acbe6d8494a774776c9f882036b753e"}
Nov 02 17:53:44.434322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.434228    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"]
Nov 02 17:53:44.439455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.439428    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.440802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.440766    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"]
Nov 02 17:53:44.447067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.447037    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkcr\" (UniqueName: \"kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.447213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.447162    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.447323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.447302    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.548972 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.548934    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkcr\" (UniqueName: \"kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.549170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.549032    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.549170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.549152    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.549703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.549680    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.551822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.551801    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.557711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.557689    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkcr\" (UniqueName: \"kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr\") pod \"maas-api-revoke-sub-cfbf9bc2-hmwsv\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:44.762204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:44.762167    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:45.067911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.067805    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"]
Nov 02 17:53:45.101416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.101034    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv" event={"ID":"eef542d8-9ca3-4414-85c2-6db6c1849193","Type":"ContainerStarted","Data":"2112a1de0590fbf04566362c54c58833878983f0a8179b861668f0ab1ddf95ae"}
Nov 02 17:53:45.284621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.284585    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:45.356912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.356787    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959\" (UniqueName: \"kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959\") pod \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") "
Nov 02 17:53:45.356912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.356872    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca\") pod \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") "
Nov 02 17:53:45.357254 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.357229    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca" pod "1f6be90a-cf4c-4ec4-8190-03028b6f0fea" (UID: "1f6be90a-cf4c-4ec4-8190-03028b6f0fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:45.357499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.357479    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:45.359312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.359289    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959" pod "1f6be90a-cf4c-4ec4-8190-03028b6f0fea" (UID: "1f6be90a-cf4c-4ec4-8190-03028b6f0fea"). InnerVolumeSpecName "kube-api-access-52959". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:45.458139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.458103    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca\") pod \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\" (UID: \"1f6be90a-cf4c-4ec4-8190-03028b6f0fea\") "
Nov 02 17:53:45.458626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.458605    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52959\" (UniqueName: \"kubernetes.io/projected/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-kube-api-access-52959\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:45.460303 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.460282    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca" pod "1f6be90a-cf4c-4ec4-8190-03028b6f0fea" (UID: "1f6be90a-cf4c-4ec4-8190-03028b6f0fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:45.564888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:45.559932    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f6be90a-cf4c-4ec4-8190-03028b6f0fea-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:46.111034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:46.110981    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv" event={"ID":"eef542d8-9ca3-4414-85c2-6db6c1849193","Type":"ContainerStarted","Data":"7eba7512cccdb1d71280a2f4d7a8c5ca8e5889345b39c2bb56b7d6f82b1c9eba"}
Nov 02 17:53:46.113471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:46.113441    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6" event={"ID":"1f6be90a-cf4c-4ec4-8190-03028b6f0fea","Type":"ContainerDied","Data":"9b1f9e24c1572776bfb200f873742a10452cb3139487fc753b4f21dd962e1718"}
Nov 02 17:53:46.113636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:46.113491    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1f9e24c1572776bfb200f873742a10452cb3139487fc753b4f21dd962e1718"
Nov 02 17:53:46.113636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:46.113539    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"
Nov 02 17:53:46.129160 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:46.129112    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv" podStartSLOduration=2.129097394 podStartE2EDuration="2.129097394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:46.126712643 +0000 UTC m=+1238.706122040" watchObservedRunningTime="2026-11-02 17:53:46.129097394 +0000 UTC m=+1238.708506779"
Nov 02 17:53:47.050866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:47.040755    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"]
Nov 02 17:53:47.050866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:47.050588    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1683322-94jtq"]
Nov 02 17:53:47.159429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:47.159071    5042 generic.go:356] "Generic (PLEG): container finished" podID="eef542d8-9ca3-4414-85c2-6db6c1849193" containerID="7eba7512cccdb1d71280a2f4d7a8c5ca8e5889345b39c2bb56b7d6f82b1c9eba" exitCode=0
Nov 02 17:53:47.163216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:47.159569    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv" event={"ID":"eef542d8-9ca3-4414-85c2-6db6c1849193","Type":"ContainerDied","Data":"7eba7512cccdb1d71280a2f4d7a8c5ca8e5889345b39c2bb56b7d6f82b1c9eba"}
Nov 02 17:53:47.685947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:47.680302    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6dd0e5d-081f-4826-a542-c7388276bf55" path="/var/lib/kubelet/pods/f6dd0e5d-081f-4826-a542-c7388276bf55/volumes"
Nov 02 17:53:48.475556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.475519    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"]
Nov 02 17:53:48.483018 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.482958    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1f6be90a-cf4c-4ec4-8190-03028b6f0fea" containerName="revoke-keys"
Nov 02 17:53:48.483253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.483231    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6be90a-cf4c-4ec4-8190-03028b6f0fea" containerName="revoke-keys"
Nov 02 17:53:48.483610 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.483588    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6be90a-cf4c-4ec4-8190-03028b6f0fea" containerName="revoke-keys"
Nov 02 17:53:48.656004 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.655783    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.711896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.675053    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e198eb-openshift-default-dockercfg-7hf67\""
Nov 02 17:53:48.720455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.720359    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"]
Nov 02 17:53:48.847924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.847879    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6nh\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.848389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.848360    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.848554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.848538    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.871869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.848717    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.872632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.872597    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.873058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.873034    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.873246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.873227    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.873380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.873353    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.873552 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.873534    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.946643 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:53:48.946104    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:53:48.975795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.975752    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6nh\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.976250 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.976221    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.976375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.976360    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.976528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.976511    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.982873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.976635    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.983622 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.983595    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.983794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.983774    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.983955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.983938    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:48.984110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:48.984094    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.004665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.004583    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.005638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.005593    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.006058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.006038    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.006754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.006734    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.007134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.007115    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.027876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.023523    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.032425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.032358    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.049458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.049177    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.068863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.066610    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6nh\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh\") pod \"e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") " pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.181195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.181145    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:53:49.214554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.214462    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:49.427866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.417537    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:49.708864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.707105    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:53:49.780707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:49.765958    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.253644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.217306    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:53:50.254173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.239939    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.258164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.256199    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:53:50.265863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.256937    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:50.566141 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.559746    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca\") pod \"eef542d8-9ca3-4414-85c2-6db6c1849193\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") "
Nov 02 17:53:50.566765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.566721    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr\" (UniqueName: \"kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr\") pod \"eef542d8-9ca3-4414-85c2-6db6c1849193\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") "
Nov 02 17:53:50.567155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.567133    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca\") pod \"eef542d8-9ca3-4414-85c2-6db6c1849193\" (UID: \"eef542d8-9ca3-4414-85c2-6db6c1849193\") "
Nov 02 17:53:50.591581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.563067    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.591581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.562527    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:53:50.623023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.609361    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:53:50.623023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.609444    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.639943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.638248    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca" pod "eef542d8-9ca3-4414-85c2-6db6c1849193" (UID: "eef542d8-9ca3-4414-85c2-6db6c1849193"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:50.639943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.639749    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:53:50.642812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.642781    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:53:50.743868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.727152    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:50.743868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.727225    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.743868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.727619    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:53:50.915684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.787155    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.915684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.789676    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:50.915684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.854217    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:53:50.925865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:50.917050    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca" pod "eef542d8-9ca3-4414-85c2-6db6c1849193" (UID: "eef542d8-9ca3-4414-85c2-6db6c1849193"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:51.059864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.015453    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:53:51.059864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.018459    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eef542d8-9ca3-4414-85c2-6db6c1849193-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:51.286867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.257902    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.286867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.284860    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.286867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.285443    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:53:51.436862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.414069    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr" pod "eef542d8-9ca3-4414-85c2-6db6c1849193" (UID: "eef542d8-9ca3-4414-85c2-6db6c1849193"). InnerVolumeSpecName "kube-api-access-4rkcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:51.436862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.423398    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"
Nov 02 17:53:51.437667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.437635    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv" event={"ID":"eef542d8-9ca3-4414-85c2-6db6c1849193","Type":"ContainerDied","Data":"2112a1de0590fbf04566362c54c58833878983f0a8179b861668f0ab1ddf95ae"}
Nov 02 17:53:51.438170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.438147    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2112a1de0590fbf04566362c54c58833878983f0a8179b861668f0ab1ddf95ae"
Nov 02 17:53:51.462987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.461039    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rkcr\" (UniqueName: \"kubernetes.io/projected/eef542d8-9ca3-4414-85c2-6db6c1849193-kube-api-access-4rkcr\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:51.483905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.468093    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:51.483905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.468195    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:51.483905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.468262    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:51.548914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.539207    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"]
Nov 02 17:53:51.571989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.571941    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.617469 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.599628    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.671290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.617749    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.744912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.728379    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.780437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.780407    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:53:51.780803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.780784    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:53:51.878920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.858161    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:51.983867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.959243    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:53:51.983867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:51.959309    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.228137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.228088    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.283346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.283298    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.286986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.286940    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.287563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.287536    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:53:52.452248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.452211    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" event={"ID":"bac100f1-0d8b-471c-9331-d2d5affd8dfd","Type":"ContainerStarted","Data":"47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"}
Nov 02 17:53:52.452544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.452502    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" event={"ID":"bac100f1-0d8b-471c-9331-d2d5affd8dfd","Type":"ContainerStarted","Data":"4f578aa033febcf3bba457af7cb6a2ba4b4904b8eb37088838a9c9e4b07c6668"}
Nov 02 17:53:52.585866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.572741    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podStartSLOduration=4.572719426 podStartE2EDuration="4.572719426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:52.487139386 +0000 UTC m=+1245.066548772" watchObservedRunningTime="2026-11-02 17:53:52.572719426 +0000 UTC m=+1245.152128812"
Nov 02 17:53:52.671330 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.671283    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.671979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.671944    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.852957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.851940    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:52.885864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.859132    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:53:52.916264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:52.916228    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:53:53.135949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.135859    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:53:53.183328 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.183299    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:53:53.183573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.183557    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:53:53.229561 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.229534    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:53.229859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.229825    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:53:53.243781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.243750    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:53:53.244055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.244024    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 17:53:53.275549 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:53.266148    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:53:54.226867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:54.226154    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:54.513197 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:54.513099    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"]
Nov 02 17:53:54.513667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:54.513635    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd" podUID="8376352d-916c-4ebe-b4e0-01317cd52091" containerName="payload-pre-processing" containerID="cri-o://c05021a50d16ee36692867e7b3680f86f2828615d3a1a323c2c135f95606c8d0" gracePeriod=30
Nov 02 17:53:54.569949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:54.569928    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:54.649144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:54.649111    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:53:55.770646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:55.761941    5042 generic.go:356] "Generic (PLEG): container finished" podID="8376352d-916c-4ebe-b4e0-01317cd52091" containerID="c05021a50d16ee36692867e7b3680f86f2828615d3a1a323c2c135f95606c8d0" exitCode=0
Nov 02 17:53:55.770646 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:55.763278    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd" event={"ID":"8376352d-916c-4ebe-b4e0-01317cd52091","Type":"ContainerDied","Data":"c05021a50d16ee36692867e7b3680f86f2828615d3a1a323c2c135f95606c8d0"}
Nov 02 17:53:55.946697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:55.946668    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:56.028749 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.028708    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume\") pod \"8376352d-916c-4ebe-b4e0-01317cd52091\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") "
Nov 02 17:53:56.029716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.029691    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7\" (UniqueName: \"kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7\") pod \"8376352d-916c-4ebe-b4e0-01317cd52091\" (UID: \"8376352d-916c-4ebe-b4e0-01317cd52091\") "
Nov 02 17:53:56.030752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.029397    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume" pod "8376352d-916c-4ebe-b4e0-01317cd52091" (UID: "8376352d-916c-4ebe-b4e0-01317cd52091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:56.059038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.058976    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7" pod "8376352d-916c-4ebe-b4e0-01317cd52091" (UID: "8376352d-916c-4ebe-b4e0-01317cd52091"). InnerVolumeSpecName "kube-api-access-k5px7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:56.060896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.059885    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"]
Nov 02 17:53:56.061097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.061079    5042 cpu_manager.go:414] "containerMap: removing container" podUID="eef542d8-9ca3-4414-85c2-6db6c1849193" containerName="revoke-keys"
Nov 02 17:53:56.063235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.061170    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef542d8-9ca3-4414-85c2-6db6c1849193" containerName="revoke-keys"
Nov 02 17:53:56.063235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.061193    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8376352d-916c-4ebe-b4e0-01317cd52091" containerName="payload-pre-processing"
Nov 02 17:53:56.063235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.061203    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8376352d-916c-4ebe-b4e0-01317cd52091" containerName="payload-pre-processing"
Nov 02 17:53:56.063235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.061375    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef542d8-9ca3-4414-85c2-6db6c1849193" containerName="revoke-keys"
Nov 02 17:53:56.063235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.061393    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8376352d-916c-4ebe-b4e0-01317cd52091" containerName="payload-pre-processing"
Nov 02 17:53:56.068079 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.068059    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.070201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.070171    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"]
Nov 02 17:53:56.084147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.078127    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:56.084147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.078823    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:53:56.150309 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.150214    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:53:56.152038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.151288    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:53:56.152225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.151534    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.152373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.152356    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.152681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.152661    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdz9z\" (UniqueName: \"kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.153016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.152997    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5px7\" (UniqueName: \"kubernetes.io/projected/8376352d-916c-4ebe-b4e0-01317cd52091-kube-api-access-k5px7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:56.153142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.153128    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8376352d-916c-4ebe-b4e0-01317cd52091-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:53:56.255816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.254125    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdz9z\" (UniqueName: \"kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.255816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.254229    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.255816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.254268    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.256606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.256537    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.258611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.258566    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.267487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.267454    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdz9z\" (UniqueName: \"kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z\") pod \"maas-api-revoke-sub-5f3d6a1a-jpklq\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:56.470369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:56.470333    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:53:57.073929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.064273    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"
Nov 02 17:53:57.105541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.082160    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd" event={"ID":"8376352d-916c-4ebe-b4e0-01317cd52091","Type":"ContainerDied","Data":"b6665c8f7a7a39fb9ba44783322c7ce558476018fa8e7d1f702fe6010cdad611"}
Nov 02 17:53:57.105902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.105876    5042 scope.go:122] "RemoveContainer" containerID="c05021a50d16ee36692867e7b3680f86f2828615d3a1a323c2c135f95606c8d0"
Nov 02 17:53:57.174644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.172814    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"]
Nov 02 17:53:57.214454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.203490    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eda8e2-5ccd55b8ff-ktjtd"]
Nov 02 17:53:57.214454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.203555    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"]
Nov 02 17:53:57.551884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.550883    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8376352d-916c-4ebe-b4e0-01317cd52091" path="/var/lib/kubelet/pods/8376352d-916c-4ebe-b4e0-01317cd52091/volumes"
Nov 02 17:53:57.612078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.612038    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"]
Nov 02 17:53:57.632809 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.626726    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"]
Nov 02 17:53:57.632809 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.626899    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.740620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.738274    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.740620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.738471    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfdd\" (UniqueName: \"kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.740620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.738631    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.840239 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.840136    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfdd\" (UniqueName: \"kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.840442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.840267    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.840442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.840373    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.841908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.841870    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.845028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.845002    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.851122 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.851095    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfdd\" (UniqueName: \"kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd\") pod \"maas-api-revoke-sub-5c968bd4-jvhz4\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:57.947786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:57.947744    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:53:58.080761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:58.080710    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq" event={"ID":"52b0d206-cb9b-4817-a081-26dd624e3f82","Type":"ContainerStarted","Data":"c5d6e3a9a42a16f9c9dba875fc48a9bb027678abc15d802b084905e95d982314"}
Nov 02 17:53:58.081236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:58.080772    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq" event={"ID":"52b0d206-cb9b-4817-a081-26dd624e3f82","Type":"ContainerStarted","Data":"bc82c618d6b3dc0da66098ed3e1bf2eb329f5b0a88232ea9da752b7218d50ad3"}
Nov 02 17:53:58.281270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:58.281222    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq" podStartSLOduration=3.281205945 podStartE2EDuration="3.281205945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:58.119122282 +0000 UTC m=+1250.698531667" watchObservedRunningTime="2026-11-02 17:53:58.281205945 +0000 UTC m=+1250.860615331"
Nov 02 17:53:58.283046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:58.283019    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"]
Nov 02 17:53:58.283913 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:53:58.283886    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c76f28_8ec8_4347_8700_1294ac44f4b5.slice/crio-cb3693e48814c7efe6b39a65a9a208e68544fda7e77c5467cb844c7f63173968 WatchSource:0}: Error finding container cb3693e48814c7efe6b39a65a9a208e68544fda7e77c5467cb844c7f63173968: Status 404 returned error can't find the container with id cb3693e48814c7efe6b39a65a9a208e68544fda7e77c5467cb844c7f63173968
Nov 02 17:53:59.095007 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:59.094067    5042 generic.go:356] "Generic (PLEG): container finished" podID="52b0d206-cb9b-4817-a081-26dd624e3f82" containerID="c5d6e3a9a42a16f9c9dba875fc48a9bb027678abc15d802b084905e95d982314" exitCode=0
Nov 02 17:53:59.095007 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:59.094209    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq" event={"ID":"52b0d206-cb9b-4817-a081-26dd624e3f82","Type":"ContainerDied","Data":"c5d6e3a9a42a16f9c9dba875fc48a9bb027678abc15d802b084905e95d982314"}
Nov 02 17:53:59.097537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:59.097506    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4" event={"ID":"e8c76f28-8ec8-4347-8700-1294ac44f4b5","Type":"ContainerStarted","Data":"880131df7a47652ab9335230be657f288ed619e6eff2523f79a5344eb5e44c2d"}
Nov 02 17:53:59.097696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:59.097565    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4" event={"ID":"e8c76f28-8ec8-4347-8700-1294ac44f4b5","Type":"ContainerStarted","Data":"cb3693e48814c7efe6b39a65a9a208e68544fda7e77c5467cb844c7f63173968"}
Nov 02 17:53:59.142002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:53:59.141943    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4" podStartSLOduration=2.141922137 podStartE2EDuration="2.141922137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:59.138306549 +0000 UTC m=+1251.717715934" watchObservedRunningTime="2026-11-02 17:53:59.141922137 +0000 UTC m=+1251.721331524"
Nov 02 17:54:00.130349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.129681    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4" event={"ID":"e8c76f28-8ec8-4347-8700-1294ac44f4b5","Type":"ContainerDied","Data":"880131df7a47652ab9335230be657f288ed619e6eff2523f79a5344eb5e44c2d"}
Nov 02 17:54:00.130956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.130448    5042 generic.go:356] "Generic (PLEG): container finished" podID="e8c76f28-8ec8-4347-8700-1294ac44f4b5" containerID="880131df7a47652ab9335230be657f288ed619e6eff2523f79a5344eb5e44c2d" exitCode=0
Nov 02 17:54:00.574832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.574776    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:54:00.640474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.640432    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca\") pod \"52b0d206-cb9b-4817-a081-26dd624e3f82\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") "
Nov 02 17:54:00.640474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.640476    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z\" (UniqueName: \"kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z\") pod \"52b0d206-cb9b-4817-a081-26dd624e3f82\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") "
Nov 02 17:54:00.640726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.640604    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca\") pod \"52b0d206-cb9b-4817-a081-26dd624e3f82\" (UID: \"52b0d206-cb9b-4817-a081-26dd624e3f82\") "
Nov 02 17:54:00.641481 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.641452    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca" pod "52b0d206-cb9b-4817-a081-26dd624e3f82" (UID: "52b0d206-cb9b-4817-a081-26dd624e3f82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:00.643126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.643095    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z" pod "52b0d206-cb9b-4817-a081-26dd624e3f82" (UID: "52b0d206-cb9b-4817-a081-26dd624e3f82"). InnerVolumeSpecName "kube-api-access-qdz9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:00.643967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.643939    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca" pod "52b0d206-cb9b-4817-a081-26dd624e3f82" (UID: "52b0d206-cb9b-4817-a081-26dd624e3f82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:00.743325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.743282    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.743325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.743328    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b0d206-cb9b-4817-a081-26dd624e3f82-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:00.743575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:00.743344    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdz9z\" (UniqueName: \"kubernetes.io/projected/52b0d206-cb9b-4817-a081-26dd624e3f82-kube-api-access-qdz9z\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:01.142406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.142373    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"
Nov 02 17:54:01.143888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.143856    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq" event={"ID":"52b0d206-cb9b-4817-a081-26dd624e3f82","Type":"ContainerDied","Data":"bc82c618d6b3dc0da66098ed3e1bf2eb329f5b0a88232ea9da752b7218d50ad3"}
Nov 02 17:54:01.144070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.144055    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc82c618d6b3dc0da66098ed3e1bf2eb329f5b0a88232ea9da752b7218d50ad3"
Nov 02 17:54:01.380592 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.380558    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:54:01.455615 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.455577    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca\") pod \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") "
Nov 02 17:54:01.455822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.455772    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca\") pod \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") "
Nov 02 17:54:01.455946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.455903    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd\" (UniqueName: \"kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd\") pod \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\" (UID: \"e8c76f28-8ec8-4347-8700-1294ac44f4b5\") "
Nov 02 17:54:01.456194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.456163    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca" pod "e8c76f28-8ec8-4347-8700-1294ac44f4b5" (UID: "e8c76f28-8ec8-4347-8700-1294ac44f4b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:01.456599 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.456576    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:01.458285 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.458244    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd" pod "e8c76f28-8ec8-4347-8700-1294ac44f4b5" (UID: "e8c76f28-8ec8-4347-8700-1294ac44f4b5"). InnerVolumeSpecName "kube-api-access-hdfdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:01.459048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.459030    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca" pod "e8c76f28-8ec8-4347-8700-1294ac44f4b5" (UID: "e8c76f28-8ec8-4347-8700-1294ac44f4b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:01.557943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.557911    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfdd\" (UniqueName: \"kubernetes.io/projected/e8c76f28-8ec8-4347-8700-1294ac44f4b5-kube-api-access-hdfdd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:01.557943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:01.557945    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8c76f28-8ec8-4347-8700-1294ac44f4b5-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:02.251861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:02.249372    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4" event={"ID":"e8c76f28-8ec8-4347-8700-1294ac44f4b5","Type":"ContainerDied","Data":"cb3693e48814c7efe6b39a65a9a208e68544fda7e77c5467cb844c7f63173968"}
Nov 02 17:54:02.251861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:02.249438    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb3693e48814c7efe6b39a65a9a208e68544fda7e77c5467cb844c7f63173968"
Nov 02 17:54:02.251861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:02.249505    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"
Nov 02 17:54:03.279256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.279213    5042 generic.go:356] "Generic (PLEG): container finished" podID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerID="846543a8927fd97c3dea5d16376205e81340f4c02cc45eb3d8233061700dceae" exitCode=0
Nov 02 17:54:03.279830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.279306    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" event={"ID":"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6","Type":"ContainerDied","Data":"846543a8927fd97c3dea5d16376205e81340f4c02cc45eb3d8233061700dceae"}
Nov 02 17:54:03.280922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.280828    5042 scope.go:122] "RemoveContainer" containerID="846543a8927fd97c3dea5d16376205e81340f4c02cc45eb3d8233061700dceae"
Nov 02 17:54:03.282815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.282771    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:03.282964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.282916    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:03.282964 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.282946    5042 generic.go:356] "Generic (PLEG): container finished" podID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerID="2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066" exitCode=0
Nov 02 17:54:03.283075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.282967    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:03.283075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.283032    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" event={"ID":"24c61ce6-6b5f-4c3c-a51f-506b49ba072c","Type":"ContainerDied","Data":"2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"}
Nov 02 17:54:03.283608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.283532    5042 scope.go:122] "RemoveContainer" containerID="2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"
Nov 02 17:54:03.340245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.338606    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:03.340245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.339166    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:03.340245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:03.339357    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:04.330198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.330160    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" event={"ID":"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6","Type":"ContainerStarted","Data":"a5648131ef4d40cf6becb7e796645ae88ec4a724de757afdfc2a6507748026ad"}
Nov 02 17:54:04.338659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.338627    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" event={"ID":"24c61ce6-6b5f-4c3c-a51f-506b49ba072c","Type":"ContainerStarted","Data":"e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd"}
Nov 02 17:54:04.878873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.878202    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:54:04.899862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.895342    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 17:54:04.899862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.895403    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 17:54:04.954802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.954764    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:54:04.955940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.955908    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 17:54:04.956092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:04.955983    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 17:54:05.880256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:05.880216    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 17:54:05.880914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:05.880877    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 17:54:05.954928 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:05.954889    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 17:54:05.955110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:05.954966    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 17:54:06.911055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:06.911020    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 17:54:06.928229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:06.928187    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 17:54:06.955212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:06.955184    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 17:54:06.955391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:06.955231    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 17:54:07.109278 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.109237    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:54:07.862946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.862872    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"]
Nov 02 17:54:07.863767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.863747    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e8c76f28-8ec8-4347-8700-1294ac44f4b5" containerName="revoke-keys"
Nov 02 17:54:07.863883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.863769    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c76f28-8ec8-4347-8700-1294ac44f4b5" containerName="revoke-keys"
Nov 02 17:54:07.863883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.863794    5042 cpu_manager.go:414] "containerMap: removing container" podUID="52b0d206-cb9b-4817-a081-26dd624e3f82" containerName="revoke-keys"
Nov 02 17:54:07.863883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.863801    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b0d206-cb9b-4817-a081-26dd624e3f82" containerName="revoke-keys"
Nov 02 17:54:07.864057 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.863980    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c76f28-8ec8-4347-8700-1294ac44f4b5" containerName="revoke-keys"
Nov 02 17:54:07.864057 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.863996    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b0d206-cb9b-4817-a081-26dd624e3f82" containerName="revoke-keys"
Nov 02 17:54:07.870542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.870518    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"]
Nov 02 17:54:07.885873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.871899    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:07.902949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.877191    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 17:54:07.902949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.895428    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 17:54:07.902949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.902889    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dfd356-dockercfg-bdqt7\""
Nov 02 17:54:07.915247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.915213    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dfd356\""
Nov 02 17:54:07.955017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.954966    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 17:54:07.955296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.955262    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 17:54:07.956711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.956685    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"]
Nov 02 17:54:07.956947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.956932    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"]
Nov 02 17:54:07.957097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.957072    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"]
Nov 02 17:54:07.957272 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.957246    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:07.967329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.967300    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:07.967768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.967742    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfs7t\" (UniqueName: \"kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t\") pod \"payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:07.969675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.969645    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"]
Nov 02 17:54:07.969809 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:07.969794    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.021251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.021210    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:54:08.035359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.022269    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:54:08.035359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.034087    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:08.035359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.034176    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:54:08.069491 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069463    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.069665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069511    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfs7t\" (UniqueName: \"kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t\") pod \"payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:08.069665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069563    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6f64\" (UniqueName: \"kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64\") pod \"payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.069665 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069607    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.069862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069684    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.069862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069714    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.069862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.069757    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:08.070530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.070506    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:08.148367 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.147507    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfs7t\" (UniqueName: \"kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t\") pod \"payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:08.171547 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.171503    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.171754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.171724    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.171817 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.171786    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6f64\" (UniqueName: \"kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64\") pod \"payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.179020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.171831    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.179020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.171958    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.185891 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.185855    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.193948 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.186631    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.198869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.198654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.237870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.227300    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6f64\" (UniqueName: \"kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64\") pod \"payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.255862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.255566    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf\") pod \"maas-api-revoke-sub-df6c2497-89h6x\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.258413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.258388    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:08.353976 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.352414    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:08.364789 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.364760    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:08.936917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.914259    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 17:54:08.936917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.914335    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 17:54:08.970787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.970503    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 17:54:08.970787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:08.970547    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 17:54:09.163256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.163216    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"]
Nov 02 17:54:09.165036 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:09.165012    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3830c9d3_c656_4b15_9757_474a158450d8.slice/crio-db74aa6656017187e0bd6cccb3f40ad6fada5e7d940a2b978491720c3d1fb338 WatchSource:0}: Error finding container db74aa6656017187e0bd6cccb3f40ad6fada5e7d940a2b978491720c3d1fb338: Status 404 returned error can't find the container with id db74aa6656017187e0bd6cccb3f40ad6fada5e7d940a2b978491720c3d1fb338
Nov 02 17:54:09.524147 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:09.524104    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a22c7b_d696_4a22_9eb0_ca2d9e0d2659.slice/crio-2f6e48b00090d9fc1b454f66fb22db9af20ff9d31e76a6afb7d9d8f8999d054f WatchSource:0}: Error finding container 2f6e48b00090d9fc1b454f66fb22db9af20ff9d31e76a6afb7d9d8f8999d054f: Status 404 returned error can't find the container with id 2f6e48b00090d9fc1b454f66fb22db9af20ff9d31e76a6afb7d9d8f8999d054f
Nov 02 17:54:09.526714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.526685    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"]
Nov 02 17:54:09.536207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.533344    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"]
Nov 02 17:54:09.663672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.663642    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx" event={"ID":"593f187e-2b2d-4e4d-9762-332be46734c1","Type":"ContainerStarted","Data":"b9cd3fef89fcc4bb238bb7d4ce092d09f050b66a4916b843181dad2372530959"}
Nov 02 17:54:09.665587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.665563    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm" event={"ID":"3830c9d3-c656-4b15-9757-474a158450d8","Type":"ContainerStarted","Data":"b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440"}
Nov 02 17:54:09.665719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.665604    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm" event={"ID":"3830c9d3-c656-4b15-9757-474a158450d8","Type":"ContainerStarted","Data":"db74aa6656017187e0bd6cccb3f40ad6fada5e7d940a2b978491720c3d1fb338"}
Nov 02 17:54:09.665782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.665747    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:09.667471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.667449    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x" event={"ID":"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659","Type":"ContainerStarted","Data":"2f6e48b00090d9fc1b454f66fb22db9af20ff9d31e76a6afb7d9d8f8999d054f"}
Nov 02 17:54:09.739866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.734781    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm" podStartSLOduration=2.73475781 podStartE2EDuration="2.73475781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:09.690346215 +0000 UTC m=+1262.269755602" watchObservedRunningTime="2026-11-02 17:54:09.73475781 +0000 UTC m=+1262.314167197"
Nov 02 17:54:09.884823 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.884789    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 17:54:09.905867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.885082    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 17:54:09.957115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.956529    5042 patch_prober.go:29] interesting pod/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 17:54:09.957115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.956603    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 17:54:09.966056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.966029    5042 scope.go:122] "RemoveContainer" containerID="c4eb3fce5ca8b08f4c5331f59d2a6f9f9cf809bb5fec5e3fb2c7bb157e13f70c"
Nov 02 17:54:09.985015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:09.984948    5042 scope.go:122] "RemoveContainer" containerID="3df0cdee42d427ea3a19908d0d3e0e3fac430578ea8f1d3b473a2402d47c2c6c"
Nov 02 17:54:10.681348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.681308    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx" event={"ID":"593f187e-2b2d-4e4d-9762-332be46734c1","Type":"ContainerStarted","Data":"4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8"}
Nov 02 17:54:10.681541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.681419    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:10.685947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.685915    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x" event={"ID":"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659","Type":"ContainerStarted","Data":"7e9285352efab342ade7e8f729f4a233ae7d4fe3bc4a5ed09a8eddc84acaee46"}
Nov 02 17:54:10.706654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.706601    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx" podStartSLOduration=3.706584253 podStartE2EDuration="3.706584253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:10.702942689 +0000 UTC m=+1263.282352105" watchObservedRunningTime="2026-11-02 17:54:10.706584253 +0000 UTC m=+1263.285993639"
Nov 02 17:54:10.727374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.727321    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x" podStartSLOduration=3.727306957 podStartE2EDuration="3.727306957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:10.723743001 +0000 UTC m=+1263.303152388" watchObservedRunningTime="2026-11-02 17:54:10.727306957 +0000 UTC m=+1263.306716420"
Nov 02 17:54:10.828626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.821006    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:54:10.883819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.883785    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:54:10.887244 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.887217    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:54:10.962458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:10.962426    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:54:11.695373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:11.695343    5042 generic.go:356] "Generic (PLEG): container finished" podID="34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" containerID="7e9285352efab342ade7e8f729f4a233ae7d4fe3bc4a5ed09a8eddc84acaee46" exitCode=0
Nov 02 17:54:11.695593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:11.695437    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x" event={"ID":"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659","Type":"ContainerDied","Data":"7e9285352efab342ade7e8f729f4a233ae7d4fe3bc4a5ed09a8eddc84acaee46"}
Nov 02 17:54:11.697414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:11.697393    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:54:12.428968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.428804    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"]
Nov 02 17:54:12.438810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.437957    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"]
Nov 02 17:54:12.438810 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.438072    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.535468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.535435    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.535780 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.535755    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.536541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.536515    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvjs\" (UniqueName: \"kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.640368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.640333    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.640583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.640383    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.641028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.640998    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvjs\" (UniqueName: \"kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.641163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.641090    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.643088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.643063    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.651900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.651871    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvjs\" (UniqueName: \"kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs\") pod \"maas-api-revoke-sub-e09afa86-trgvk\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.850644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.850623    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:12.883390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.883347    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:12.945602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.945513    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca\") pod \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") "
Nov 02 17:54:12.945602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.945577    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf\") pod \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") "
Nov 02 17:54:12.945901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.945819    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca\") pod \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\" (UID: \"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659\") "
Nov 02 17:54:12.947106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.947027    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca" pod "34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" (UID: "34a22c7b-d696-4a22-9eb0-ca2d9e0d2659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:12.948590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.948558    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca" pod "34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" (UID: "34a22c7b-d696-4a22-9eb0-ca2d9e0d2659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:12.950711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:12.950689    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf" pod "34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" (UID: "34a22c7b-d696-4a22-9eb0-ca2d9e0d2659"). InnerVolumeSpecName "kube-api-access-5g6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:13.048059 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.047974    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:13.048059 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.048004    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g6tf\" (UniqueName: \"kubernetes.io/projected/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-kube-api-access-5g6tf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:13.048059 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.048013    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:13.145304 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.145274    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"]
Nov 02 17:54:13.146017 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:13.145989    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65dba72_19b0_41ce_b682_87c57cf2d0d3.slice/crio-d3172c83566b9781d3ab9445720a454ea484ef35f8b4ef1d35f3d1c9e7e9a0cb WatchSource:0}: Error finding container d3172c83566b9781d3ab9445720a454ea484ef35f8b4ef1d35f3d1c9e7e9a0cb: Status 404 returned error can't find the container with id d3172c83566b9781d3ab9445720a454ea484ef35f8b4ef1d35f3d1c9e7e9a0cb
Nov 02 17:54:13.713311 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.713277    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"
Nov 02 17:54:13.713775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.713299    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x" event={"ID":"34a22c7b-d696-4a22-9eb0-ca2d9e0d2659","Type":"ContainerDied","Data":"2f6e48b00090d9fc1b454f66fb22db9af20ff9d31e76a6afb7d9d8f8999d054f"}
Nov 02 17:54:13.713775 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.713371    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f6e48b00090d9fc1b454f66fb22db9af20ff9d31e76a6afb7d9d8f8999d054f"
Nov 02 17:54:13.716898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.716864    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk" event={"ID":"d65dba72-19b0-41ce-b682-87c57cf2d0d3","Type":"ContainerStarted","Data":"cfae14eb9d4cfa63754ad0888d626fd46170ad445c5498b7c8231343ef2d1cb5"}
Nov 02 17:54:13.717061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.716921    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk" event={"ID":"d65dba72-19b0-41ce-b682-87c57cf2d0d3","Type":"ContainerStarted","Data":"d3172c83566b9781d3ab9445720a454ea484ef35f8b4ef1d35f3d1c9e7e9a0cb"}
Nov 02 17:54:13.740991 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:13.740891    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk" podStartSLOduration=1.740829621 podStartE2EDuration="1.740829621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:13.737724024 +0000 UTC m=+1266.317133408" watchObservedRunningTime="2026-11-02 17:54:13.740829621 +0000 UTC m=+1266.320239047"
Nov 02 17:54:14.345176 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:14.345143    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65dba72_19b0_41ce_b682_87c57cf2d0d3.slice/crio-conmon-cfae14eb9d4cfa63754ad0888d626fd46170ad445c5498b7c8231343ef2d1cb5.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 17:54:14.727496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:14.727469    5042 generic.go:356] "Generic (PLEG): container finished" podID="d65dba72-19b0-41ce-b682-87c57cf2d0d3" containerID="cfae14eb9d4cfa63754ad0888d626fd46170ad445c5498b7c8231343ef2d1cb5" exitCode=0
Nov 02 17:54:14.727960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:14.727552    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk" event={"ID":"d65dba72-19b0-41ce-b682-87c57cf2d0d3","Type":"ContainerDied","Data":"cfae14eb9d4cfa63754ad0888d626fd46170ad445c5498b7c8231343ef2d1cb5"}
Nov 02 17:54:15.966285 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:15.966256    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:16.007661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.006446    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca\") pod \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") "
Nov 02 17:54:16.007661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.006567    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs\" (UniqueName: \"kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs\") pod \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") "
Nov 02 17:54:16.007661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.006695    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca\") pod \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\" (UID: \"d65dba72-19b0-41ce-b682-87c57cf2d0d3\") "
Nov 02 17:54:16.007661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.007645    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca" pod "d65dba72-19b0-41ce-b682-87c57cf2d0d3" (UID: "d65dba72-19b0-41ce-b682-87c57cf2d0d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:16.020044 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.020004    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca" pod "d65dba72-19b0-41ce-b682-87c57cf2d0d3" (UID: "d65dba72-19b0-41ce-b682-87c57cf2d0d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:16.020563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.020535    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs" pod "d65dba72-19b0-41ce-b682-87c57cf2d0d3" (UID: "d65dba72-19b0-41ce-b682-87c57cf2d0d3"). InnerVolumeSpecName "kube-api-access-vjvjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:16.109091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.109054    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:16.109091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.109085    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvjs\" (UniqueName: \"kubernetes.io/projected/d65dba72-19b0-41ce-b682-87c57cf2d0d3-kube-api-access-vjvjs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:16.109091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.109096    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dba72-19b0-41ce-b682-87c57cf2d0d3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:16.743892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.743824    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"
Nov 02 17:54:16.744077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.743821    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk" event={"ID":"d65dba72-19b0-41ce-b682-87c57cf2d0d3","Type":"ContainerDied","Data":"d3172c83566b9781d3ab9445720a454ea484ef35f8b4ef1d35f3d1c9e7e9a0cb"}
Nov 02 17:54:16.744077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:16.743983    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3172c83566b9781d3ab9445720a454ea484ef35f8b4ef1d35f3d1c9e7e9a0cb"
Nov 02 17:54:18.593551 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:18.593508    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:54:20.703420 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:20.703384    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:54:21.708970 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.708936    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"]
Nov 02 17:54:21.710448 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.710421    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d65dba72-19b0-41ce-b682-87c57cf2d0d3" containerName="revoke-keys"
Nov 02 17:54:21.710640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.710626    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65dba72-19b0-41ce-b682-87c57cf2d0d3" containerName="revoke-keys"
Nov 02 17:54:21.710912 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.710896    5042 cpu_manager.go:414] "containerMap: removing container" podUID="34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" containerName="revoke-keys"
Nov 02 17:54:21.711009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.710995    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" containerName="revoke-keys"
Nov 02 17:54:21.711293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.711279    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" containerName="revoke-keys"
Nov 02 17:54:21.713955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.713932    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65dba72-19b0-41ce-b682-87c57cf2d0d3" containerName="revoke-keys"
Nov 02 17:54:21.762487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.760384    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"]
Nov 02 17:54:21.762487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.760485    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:54:21.762487 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.761519    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907292 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907252    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907410    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907483    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907508    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907547    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28hv\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907585    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907629    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907677    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:21.907788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:21.907701    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.008940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.008829    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.008940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.008942    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.008988    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.009006    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.009031    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28hv\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.009055    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.009079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.009108    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.009152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.009123    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.010531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.010227    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.010531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.010339    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.010531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.010488    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.010871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.010805    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.011052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.011030    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.011816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.011792    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.012637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.012617    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.022399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.022317    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.022613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.022594    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28hv\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv\") pod \"e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") " pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.136605 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.136563    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:22.510139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.510110    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"]
Nov 02 17:54:22.516686 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.516646    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:22.516818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.516732    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:22.516818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:22.516775    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:23.044514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.044239    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" event={"ID":"fa2ee4bf-7406-4fe9-bfc1-569e0192d292","Type":"ContainerStarted","Data":"f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9"}
Nov 02 17:54:23.044514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.044338    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" event={"ID":"fa2ee4bf-7406-4fe9-bfc1-569e0192d292","Type":"ContainerStarted","Data":"2bedc43120133bdc9bd7b6894c8ab72a6b829cb13afb5d3e5f2996ece6717053"}
Nov 02 17:54:23.120870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.114754    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podStartSLOduration=2.114726891 podStartE2EDuration="2.114726891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:23.081283741 +0000 UTC m=+1275.660693128" watchObservedRunningTime="2026-11-02 17:54:23.114726891 +0000 UTC m=+1275.694136280"
Nov 02 17:54:23.204863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.203656    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:23.204863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.204232    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:23.204863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.204302    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:23.454405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.454366    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"]
Nov 02 17:54:23.461860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.461813    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.533509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.533462    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"]
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.556769    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.556944    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpzj\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557001    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557100    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557182    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557312    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557384    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557419    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.566868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.557461    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.660989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.660957    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661014    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661033    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661059    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661080    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661149    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpzj\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661180 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661177    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661247    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661306    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.661542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661417    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.662016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.661994    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.662406 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.662385    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.663101 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.663079    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.663743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.663719    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.664199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.664178    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.665145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.665124    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.678710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.678677    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.737919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.730619    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpzj\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj\") pod \"e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") " pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:23.959530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:23.959501    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:24.062160 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.062126    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"]
Nov 02 17:54:24.068317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.068282    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"]
Nov 02 17:54:24.068457 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.068416    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.082020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.080742    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:24.082020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.080866    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:54:24.082020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.080946    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:54:24.082020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.081022    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:54:24.141737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.140705    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:24.141737 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.140768    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:24.296931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.281483    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2mn\" (UniqueName: \"kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.296931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.281679    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.296931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.281924    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.406866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.395287    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.406866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.395409    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2mn\" (UniqueName: \"kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.406866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.395553    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.429863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.428580    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.470736 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.470703    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2mn\" (UniqueName: \"kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.530757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.530724    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-014ff898-nd5ft\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:24.622887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.622830    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"]
Nov 02 17:54:24.633019 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.632986    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:24.633125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.633065    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:24.633125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.633098    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:24.815827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:24.815567    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:25.132864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.132254    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" event={"ID":"d173258a-c30f-463b-8aa0-298885d64950","Type":"ContainerStarted","Data":"9362b51fa34c8dbf29dfd73277f55119a3ca6b556d063e575a6b4ab4da1eac62"}
Nov 02 17:54:25.132864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.132315    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" event={"ID":"d173258a-c30f-463b-8aa0-298885d64950","Type":"ContainerStarted","Data":"1424241ece05e806d34cb027e10bc4b1825226c85faf7d003e486e001047f539"}
Nov 02 17:54:25.140388 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.140354    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:25.140567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.140413    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:25.168795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.168730    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podStartSLOduration=2.168709594 podStartE2EDuration="2.168709594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:25.164182923 +0000 UTC m=+1277.743592310" watchObservedRunningTime="2026-11-02 17:54:25.168709594 +0000 UTC m=+1277.748118984"
Nov 02 17:54:25.234107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.234059    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"]
Nov 02 17:54:25.615514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.615479    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:54:25.615803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.615771    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:25.960181 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.960140    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:25.961907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.961631    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:25.961907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:25.961677    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:26.156096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.155997    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:26.156096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.156061    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:26.163934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.163886    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft" event={"ID":"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5","Type":"ContainerStarted","Data":"268b59ed6069f7089d12a70e5a0dd1d00844dcf4035b4e81c75d3ab1545492c7"}
Nov 02 17:54:26.164123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.163944    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft" event={"ID":"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5","Type":"ContainerStarted","Data":"080aa29622a99ca3121a90e63c865ddc944cbf0a78ca1ee3d928fe72fe8f7740"}
Nov 02 17:54:26.202063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.201996    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft" podStartSLOduration=3.201971442 podStartE2EDuration="3.201971442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:26.193330403 +0000 UTC m=+1278.772739790" watchObservedRunningTime="2026-11-02 17:54:26.201971442 +0000 UTC m=+1278.781380830"
Nov 02 17:54:26.966858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.966247    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:26.966858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:26.966308    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:27.141571 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:27.141484    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:27.141571 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:27.141567    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:27.184867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:27.179763    5042 generic.go:356] "Generic (PLEG): container finished" podID="e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" containerID="268b59ed6069f7089d12a70e5a0dd1d00844dcf4035b4e81c75d3ab1545492c7" exitCode=0
Nov 02 17:54:27.192502 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:27.191346    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft" event={"ID":"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5","Type":"ContainerDied","Data":"268b59ed6069f7089d12a70e5a0dd1d00844dcf4035b4e81c75d3ab1545492c7"}
Nov 02 17:54:28.048923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.048887    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:28.049644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.049173    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:28.140195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.140047    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:28.140195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.140115    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:28.935929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.933313    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:28.942635 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.942598    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca\") pod \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") "
Nov 02 17:54:28.942899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.942875    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn\" (UniqueName: \"kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn\") pod \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") "
Nov 02 17:54:28.943012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.942955    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca\") pod \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\" (UID: \"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5\") "
Nov 02 17:54:28.943260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.943204    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca" pod "e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" (UID: "e167f8b8-37c6-4fd3-be22-cb46c45ca6d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:28.943658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.943585    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:28.945320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.945294    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca" pod "e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" (UID: "e167f8b8-37c6-4fd3-be22-cb46c45ca6d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:28.945485 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.945458    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn" pod "e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" (UID: "e167f8b8-37c6-4fd3-be22-cb46c45ca6d5"). InnerVolumeSpecName "kube-api-access-sd2mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:28.960228 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.960191    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:28.960356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:28.960278    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:29.046866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.046446    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd2mn\" (UniqueName: \"kubernetes.io/projected/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-kube-api-access-sd2mn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:29.046866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.046484    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:29.139261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.138953    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:29.139261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.139017    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:29.343937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.343251    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft" event={"ID":"e167f8b8-37c6-4fd3-be22-cb46c45ca6d5","Type":"ContainerDied","Data":"080aa29622a99ca3121a90e63c865ddc944cbf0a78ca1ee3d928fe72fe8f7740"}
Nov 02 17:54:29.343937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.343318    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080aa29622a99ca3121a90e63c865ddc944cbf0a78ca1ee3d928fe72fe8f7740"
Nov 02 17:54:29.343937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.343394    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"
Nov 02 17:54:29.959890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.959833    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:29.960352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:29.960319    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:30.136882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:30.136812    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 17:54:30.137081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:30.136905    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 17:54:30.960604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:30.960554    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:30.961202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:30.960627    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:31.894217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:31.894175    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:31.896495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:31.896475    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:31.898862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:31.898818    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 17:54:31.967177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:31.963222    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:31.967177 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:31.963278    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:32.967326 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:32.967288    5042 patch_prober.go:29] interesting pod/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Nov 02 17:54:32.967927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:32.967882    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Nov 02 17:54:33.965261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:33.965229    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:33.965658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:33.965636    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:33.966485 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:33.966462    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 17:54:34.452695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.452598    5042 generic.go:356] "Generic (PLEG): container finished" podID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerID="47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc" exitCode=0
Nov 02 17:54:34.460962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.460916    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" event={"ID":"bac100f1-0d8b-471c-9331-d2d5affd8dfd","Type":"ContainerDied","Data":"47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"}
Nov 02 17:54:34.461545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.461523    5042 scope.go:122] "RemoveContainer" containerID="47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"
Nov 02 17:54:34.463604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.463571    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:34.463709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.463666    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:34.463752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.463716    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:34.670888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.670860    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"]
Nov 02 17:54:34.671361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:34.671332    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy" containerID="cri-o://e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd" gracePeriod=30
Nov 02 17:54:35.463585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:35.463547    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" event={"ID":"bac100f1-0d8b-471c-9331-d2d5affd8dfd","Type":"ContainerStarted","Data":"270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81"}
Nov 02 17:54:36.215137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:36.215102    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:54:36.215940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:36.215916    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:54:36.216021 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:36.215969    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 17:54:37.216871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.216276    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:54:37.216871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.216336    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 17:54:37.506381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.506293    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"]
Nov 02 17:54:37.507170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.507147    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" containerName="revoke-keys"
Nov 02 17:54:37.507291 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.507173    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" containerName="revoke-keys"
Nov 02 17:54:37.507369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.507355    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" containerName="revoke-keys"
Nov 02 17:54:37.510073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.510052    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.515438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.515413    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:54:37.515644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.515619    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"]
Nov 02 17:54:37.516545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.516523    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:54:37.539952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.539916    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586sz\" (UniqueName: \"kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.540151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.540048    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.540353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.540333    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.569669 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.569632    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:37.569904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.569700    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:54:37.642189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.642142    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.642403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.642383    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.642572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.642554    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586sz\" (UniqueName: \"kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.643597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.643568    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.645073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.645049    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.651662 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.651636    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586sz\" (UniqueName: \"kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz\") pod \"maas-api-revoke-sub-7561eeb3-lkhpr\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:37.835630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:37.835545    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:38.111813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.111707    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"]
Nov 02 17:54:38.112730 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.112687    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy" containerID="cri-o://392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b" gracePeriod=30
Nov 02 17:54:38.215992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.215395    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:54:38.215992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.215459    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 17:54:38.233099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.233068    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"]
Nov 02 17:54:38.233715 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:38.233691    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac55011b_d072_49af_ae32_89dff378f8e8.slice/crio-286baae629d716782a874faffe5708ef552aace0b2ee9cd8cae1edfd0375bac9 WatchSource:0}: Error finding container 286baae629d716782a874faffe5708ef552aace0b2ee9cd8cae1edfd0375bac9: Status 404 returned error can't find the container with id 286baae629d716782a874faffe5708ef552aace0b2ee9cd8cae1edfd0375bac9
Nov 02 17:54:38.362918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.361622    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"]
Nov 02 17:54:38.362918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.361993    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy" containerID="cri-o://19dd6d3721e32d07d8d6dfdf8b6575ea295770067860d2bb72781a73e7eaa053" gracePeriod=30
Nov 02 17:54:38.528779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:38.528739    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr" event={"ID":"ac55011b-d072-49af-ae32-89dff378f8e8","Type":"ContainerStarted","Data":"286baae629d716782a874faffe5708ef552aace0b2ee9cd8cae1edfd0375bac9"}
Nov 02 17:54:39.217918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:39.217409    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:54:39.217918 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:39.217472    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 17:54:39.540061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:39.539974    5042 generic.go:356] "Generic (PLEG): container finished" podID="ac55011b-d072-49af-ae32-89dff378f8e8" containerID="66542c6df2dbf14a430187a8008ee1a4e0d0b4411cdbe5c56c25d20e0e4ae726" exitCode=0
Nov 02 17:54:39.546083 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:39.546053    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr" event={"ID":"ac55011b-d072-49af-ae32-89dff378f8e8","Type":"ContainerDied","Data":"66542c6df2dbf14a430187a8008ee1a4e0d0b4411cdbe5c56c25d20e0e4ae726"}
Nov 02 17:54:39.567011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:39.566973    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:54:40.225106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.224538    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:54:40.225106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.224592    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Nov 02 17:54:40.247507 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.247476    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:54:40.367356 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367323    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367416    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367460    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367500    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367560    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367636    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367678    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367709    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.367732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367715    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:40.368010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367776    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data\") pod \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\" (UID: \"24c61ce6-6b5f-4c3c-a51f-506b49ba072c\") "
Nov 02 17:54:40.368010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.367984    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:40.368130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368008    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:40.368130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368013    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:40.368296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368273    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:40.368462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368440    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.368538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368471    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.368538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368490    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.368538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368505    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.368538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.368519    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.370353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.370318    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:40.370353 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.370336    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:40.370540 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.370440    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:40.370621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.370606    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt" pod "24c61ce6-6b5f-4c3c-a51f-506b49ba072c" (UID: "24c61ce6-6b5f-4c3c-a51f-506b49ba072c"). InnerVolumeSpecName "kube-api-access-mltqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:40.469638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.469603    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.469638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.469635    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.469638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.469646    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mltqt\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-kube-api-access-mltqt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.469925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.469656    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c61ce6-6b5f-4c3c-a51f-506b49ba072c-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.549638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.549548    5042 generic.go:356] "Generic (PLEG): container finished" podID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerID="e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd" exitCode=0
Nov 02 17:54:40.549638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.549607    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"
Nov 02 17:54:40.549638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.549610    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" event={"ID":"24c61ce6-6b5f-4c3c-a51f-506b49ba072c","Type":"ContainerDied","Data":"e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd"}
Nov 02 17:54:40.550204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.549667    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6" event={"ID":"24c61ce6-6b5f-4c3c-a51f-506b49ba072c","Type":"ContainerDied","Data":"bdb60edd0c0e1e5f3a968e6a7f924253c9474cdfdda8ba8fb6b92806b861bf86"}
Nov 02 17:54:40.550204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.549693    5042 scope.go:122] "RemoveContainer" containerID="e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd"
Nov 02 17:54:40.562220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.562203    5042 scope.go:122] "RemoveContainer" containerID="2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"
Nov 02 17:54:40.598132 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.598061    5042 scope.go:122] "RemoveContainer" containerID="e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd"
Nov 02 17:54:40.599433 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:40.599069    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd\": container with ID starting with e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd not found: ID does not exist" containerID="e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd"
Nov 02 17:54:40.599433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.599112    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd"} err="failed to get container status \"e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd\": rpc error: code = NotFound desc = could not find container \"e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd\": container with ID starting with e42a0125bdebd3d87d982b5f5386fa5fc2e13a2f8d52b99bb1695029a0217bcd not found: ID does not exist"
Nov 02 17:54:40.599433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.599138    5042 scope.go:122] "RemoveContainer" containerID="2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"
Nov 02 17:54:40.599918 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:40.599876    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066\": container with ID starting with 2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066 not found: ID does not exist" containerID="2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"
Nov 02 17:54:40.600092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.600068    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066"} err="failed to get container status \"2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066\": rpc error: code = NotFound desc = could not find container \"2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066\": container with ID starting with 2337c44c61c36087957013db63f4412cc827f5c43989f6fa1a4028db63f0c066 not found: ID does not exist"
Nov 02 17:54:40.619543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.619508    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"]
Nov 02 17:54:40.626755 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.626721    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4e6649-openshift-default-68f8db9665-ngbl6"]
Nov 02 17:54:40.802611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.802544    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:40.895022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.894991    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca\") pod \"ac55011b-d072-49af-ae32-89dff378f8e8\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") "
Nov 02 17:54:40.895234 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.895035    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca\") pod \"ac55011b-d072-49af-ae32-89dff378f8e8\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") "
Nov 02 17:54:40.895234 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.895165    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz\" (UniqueName: \"kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz\") pod \"ac55011b-d072-49af-ae32-89dff378f8e8\" (UID: \"ac55011b-d072-49af-ae32-89dff378f8e8\") "
Nov 02 17:54:40.896701 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.896676    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca" pod "ac55011b-d072-49af-ae32-89dff378f8e8" (UID: "ac55011b-d072-49af-ae32-89dff378f8e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:40.900760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.900726    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz" pod "ac55011b-d072-49af-ae32-89dff378f8e8" (UID: "ac55011b-d072-49af-ae32-89dff378f8e8"). InnerVolumeSpecName "kube-api-access-586sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:40.903331 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.903307    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca" pod "ac55011b-d072-49af-ae32-89dff378f8e8" (UID: "ac55011b-d072-49af-ae32-89dff378f8e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:40.997109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.997066    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.997109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.997108    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac55011b-d072-49af-ae32-89dff378f8e8-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:40.997338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:40.997125    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586sz\" (UniqueName: \"kubernetes.io/projected/ac55011b-d072-49af-ae32-89dff378f8e8-kube-api-access-586sz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:41.221871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:41.221813    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:54:41.223003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:41.222981    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:54:41.554323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:41.554237    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" path="/var/lib/kubelet/pods/24c61ce6-6b5f-4c3c-a51f-506b49ba072c/volumes"
Nov 02 17:54:41.560167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:41.560137    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"
Nov 02 17:54:41.560359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:41.560170    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr" event={"ID":"ac55011b-d072-49af-ae32-89dff378f8e8","Type":"ContainerDied","Data":"286baae629d716782a874faffe5708ef552aace0b2ee9cd8cae1edfd0375bac9"}
Nov 02 17:54:41.560359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:41.560232    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="286baae629d716782a874faffe5708ef552aace0b2ee9cd8cae1edfd0375bac9"
Nov 02 17:54:43.518805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.518778    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:54:43.579589 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.579559    5042 generic.go:356] "Generic (PLEG): container finished" podID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerID="392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b" exitCode=0
Nov 02 17:54:43.579752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.579694    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" event={"ID":"22cced10-bb3b-4b8e-8de0-f22ea4c39121","Type":"ContainerDied","Data":"392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b"}
Nov 02 17:54:43.579752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.579739    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh" event={"ID":"22cced10-bb3b-4b8e-8de0-f22ea4c39121","Type":"ContainerDied","Data":"903d598866869779ef2f521596edd879d34f0a896bd8576442e4f6ce76083805"}
Nov 02 17:54:43.579824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.579771    5042 scope.go:122] "RemoveContainer" containerID="392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b"
Nov 02 17:54:43.579979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.579965    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"
Nov 02 17:54:43.584782 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.584711    5042 generic.go:356] "Generic (PLEG): container finished" podID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerID="19dd6d3721e32d07d8d6dfdf8b6575ea295770067860d2bb72781a73e7eaa053" exitCode=0
Nov 02 17:54:43.585105 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.585058    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" event={"ID":"344a2e2e-0404-446b-844d-47e6baa07dfb","Type":"ContainerDied","Data":"19dd6d3721e32d07d8d6dfdf8b6575ea295770067860d2bb72781a73e7eaa053"}
Nov 02 17:54:43.599724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.599700    5042 scope.go:122] "RemoveContainer" containerID="392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b"
Nov 02 17:54:43.600258 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:43.600213    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b\": container with ID starting with 392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b not found: ID does not exist" containerID="392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b"
Nov 02 17:54:43.600373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.600284    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b"} err="failed to get container status \"392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b\": rpc error: code = NotFound desc = could not find container \"392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b\": container with ID starting with 392d99137a9ac4770d1e2487c0facdbf5f052115e0c187838466302a0710d29b not found: ID does not exist"
Nov 02 17:54:43.664106 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.664070    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.667477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.667451    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.667694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.667672    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.667920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.667781    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.667920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.667867    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.667920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.667907    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.668172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.667990    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.668172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.668162    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.668286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.668225    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert\") pod \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\" (UID: \"22cced10-bb3b-4b8e-8de0-f22ea4c39121\") "
Nov 02 17:54:43.669399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.669176    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.669506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.669237    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.669506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.669314    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.670688 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.670663    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:43.671124 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.671101    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:43.671215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.671202    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.680384 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.680346    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:43.680501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.680396    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "kube-api-access-vfcsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:43.680679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.680659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy" pod "22cced10-bb3b-4b8e-8de0-f22ea4c39121" (UID: "22cced10-bb3b-4b8e-8de0-f22ea4c39121"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.702953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.702929    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:54:43.771202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771168    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771197    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771202 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771208    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771217    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771227    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771237    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfcsh\" (UniqueName: \"kubernetes.io/projected/22cced10-bb3b-4b8e-8de0-f22ea4c39121-kube-api-access-vfcsh\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771247    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771257    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.771454 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.771266    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cced10-bb3b-4b8e-8de0-f22ea4c39121-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.872113 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872077    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872113 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872120    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872185    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872241    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872284    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872311    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872381    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872445    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872551    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert\") pod \"344a2e2e-0404-446b-844d-47e6baa07dfb\" (UID: \"344a2e2e-0404-446b-844d-47e6baa07dfb\") "
Nov 02 17:54:43.872585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872558    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.872791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872755    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.872863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.872818    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.873063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873039    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.873209 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873095    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:43.873541 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873519    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.873634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873549    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.873634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873567    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.873634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873581    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.873634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.873597    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344a2e2e-0404-446b-844d-47e6baa07dfb-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.875274 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.875240    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "kube-api-access-q6nvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:43.875391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.875301    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:43.875734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.875705    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:43.875819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.875735    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo" pod "344a2e2e-0404-446b-844d-47e6baa07dfb" (UID: "344a2e2e-0404-446b-844d-47e6baa07dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:43.909677 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.909642    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"]
Nov 02 17:54:43.913538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.913511    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-6p9mh"]
Nov 02 17:54:43.974631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.974543    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6nvm\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-kube-api-access-q6nvm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.974631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.974575    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.974631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.974584    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:43.974631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:43.974593    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344a2e2e-0404-446b-844d-47e6baa07dfb-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:44.597538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:44.597509    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"
Nov 02 17:54:44.598110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:44.597538    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w" event={"ID":"344a2e2e-0404-446b-844d-47e6baa07dfb","Type":"ContainerDied","Data":"7cf0730c78411dc28cae005d8b9ec1d217ae92c107850505124c3b36581a1881"}
Nov 02 17:54:44.598110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:44.597610    5042 scope.go:122] "RemoveContainer" containerID="19dd6d3721e32d07d8d6dfdf8b6575ea295770067860d2bb72781a73e7eaa053"
Nov 02 17:54:44.624966 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:44.621675    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344a2e2e_0404_446b_844d_47e6baa07dfb.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 17:54:44.633695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:44.633662    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"]
Nov 02 17:54:44.640700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:44.640667    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-m6g9w"]
Nov 02 17:54:45.544095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:45.544059    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" path="/var/lib/kubelet/pods/22cced10-bb3b-4b8e-8de0-f22ea4c39121/volumes"
Nov 02 17:54:45.544678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:45.544656    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" path="/var/lib/kubelet/pods/344a2e2e-0404-446b-844d-47e6baa07dfb/volumes"
Nov 02 17:54:48.595778 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:48.595694    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:54:50.299868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:50.296953    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:50.402865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:50.401948    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:54:50.613926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:50.612945    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:50.691863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:50.691608    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:50.869916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:50.869495    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.085804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.085765    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"]
Nov 02 17:54:51.092974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.092258    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" containerID="cri-o://270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81" gracePeriod=30
Nov 02 17:54:51.184960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.178194    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.184960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.179000    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:54:51.357865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.317980    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.357865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.321610    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.409884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.358913    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:54:51.409884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.402931    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.409884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.403546    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:54:51.556873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.556498    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.694862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.692824    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:51.862921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.860404    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:54:51.862921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.860514    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:54:51.948015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.883265    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:54:51.948015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.883567    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:54:51.948015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.930385    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:54:51.948015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:51.930444    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:52.126413 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.125065    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"]
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127683    5042 cpu_manager.go:414] "containerMap: removing container" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127712    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127740    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ac55011b-d072-49af-ae32-89dff378f8e8" containerName="revoke-keys"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127749    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac55011b-d072-49af-ae32-89dff378f8e8" containerName="revoke-keys"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127777    5042 cpu_manager.go:414] "containerMap: removing container" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127785    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127801    5042 cpu_manager.go:414] "containerMap: removing container" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127809    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127831    5042 cpu_manager.go:414] "containerMap: removing container" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.127860    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.128027    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.128039    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c61ce6-6b5f-4c3c-a51f-506b49ba072c" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.128051    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="344a2e2e-0404-446b-844d-47e6baa07dfb" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.128071    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cced10-bb3b-4b8e-8de0-f22ea4c39121" containerName="istio-proxy"
Nov 02 17:54:52.130583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.128083    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac55011b-d072-49af-ae32-89dff378f8e8" containerName="revoke-keys"
Nov 02 17:54:52.154937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.154436    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.161304 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.161228    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"]
Nov 02 17:54:52.168227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.168193    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"]
Nov 02 17:54:52.168721 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.168670    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy" containerID="cri-o://208f228e382f5dfb7e0ea2895c148fc019a7f1a7da3500b4eac7032a241b2643" gracePeriod=30
Nov 02 17:54:52.185422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.184818    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:52.185422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.185176    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:54:52.187870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.187123    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:54:52.187870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.187293    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.277098    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.277173    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.277276    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.277339    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.279471    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.279617    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcms\" (UniqueName: \"kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.319863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.279719    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.394917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.390608    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.394917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.390926    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcms\" (UniqueName: \"kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.394917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.391017    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.394917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.392789    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.394917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.393475    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:54:52.407225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.405943    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:52.416942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.407680    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:52.416942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.412660    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:52.468990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.468638    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:54:52.495761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.495727    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:54:52.496032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.496015    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:54:52.497403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.497369    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.534564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.533667    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcms\" (UniqueName: \"kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms\") pod \"maas-api-revoke-sub-b0434ab4-ds6bv\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.563887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.563823    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:54:52.643196 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.643159    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:52.960821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:52.960787    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"]
Nov 02 17:54:52.960960 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:52.960912    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b2f6bd_c213_409c_8885_517dd600d429.slice/crio-e4af85731cb0b0769b9ee2eeb460d82a9385d3f964443562a9e4853e7412a9b0 WatchSource:0}: Error finding container e4af85731cb0b0769b9ee2eeb460d82a9385d3f964443562a9e4853e7412a9b0: Status 404 returned error can't find the container with id e4af85731cb0b0769b9ee2eeb460d82a9385d3f964443562a9e4853e7412a9b0
Nov 02 17:54:53.133829 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.133796    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"]
Nov 02 17:54:53.139167 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.139150    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.153505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.153476    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"]
Nov 02 17:54:53.228900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.228739    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.228900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.228778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.229152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.229024    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5kg\" (UniqueName: \"kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.330120 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.330085    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5kg\" (UniqueName: \"kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.330313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.330280    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.330370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.330317    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.330966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.330921    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.332804 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.332781    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.340401 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.340381    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5kg\" (UniqueName: \"kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg\") pod \"maas-api-revoke-sub-3993dad0-ghhht\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.504351 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.504259    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:53.796235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.796200    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"]
Nov 02 17:54:53.797494 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:53.797464    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a872d9c_5f74_4361_a14f_1914ca50de40.slice/crio-6c815f3cb46f540b11eaee4d89b7cc772509205fd08f45570f16fb82444dd95f WatchSource:0}: Error finding container 6c815f3cb46f540b11eaee4d89b7cc772509205fd08f45570f16fb82444dd95f: Status 404 returned error can't find the container with id 6c815f3cb46f540b11eaee4d89b7cc772509205fd08f45570f16fb82444dd95f
Nov 02 17:54:53.890390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.890349    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv" event={"ID":"f3b2f6bd-c213-409c-8885-517dd600d429","Type":"ContainerStarted","Data":"ede59465a41daeeaba9d3d14ec5b4692ce152aad30498aa4558e9eacf2b81985"}
Nov 02 17:54:53.890673 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.890643    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv" event={"ID":"f3b2f6bd-c213-409c-8885-517dd600d429","Type":"ContainerStarted","Data":"e4af85731cb0b0769b9ee2eeb460d82a9385d3f964443562a9e4853e7412a9b0"}
Nov 02 17:54:53.891909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.891887    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht" event={"ID":"1a872d9c-5f74-4361-a14f-1914ca50de40","Type":"ContainerStarted","Data":"6c815f3cb46f540b11eaee4d89b7cc772509205fd08f45570f16fb82444dd95f"}
Nov 02 17:54:53.913664 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:53.913595    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv" podStartSLOduration=2.9135775219999998 podStartE2EDuration="2.913577522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:53.908972701 +0000 UTC m=+1306.488382092" watchObservedRunningTime="2026-11-02 17:54:53.913577522 +0000 UTC m=+1306.492986911"
Nov 02 17:54:54.906452 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:54.906416    5042 generic.go:356] "Generic (PLEG): container finished" podID="f3b2f6bd-c213-409c-8885-517dd600d429" containerID="ede59465a41daeeaba9d3d14ec5b4692ce152aad30498aa4558e9eacf2b81985" exitCode=0
Nov 02 17:54:54.906863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:54.906502    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv" event={"ID":"f3b2f6bd-c213-409c-8885-517dd600d429","Type":"ContainerDied","Data":"ede59465a41daeeaba9d3d14ec5b4692ce152aad30498aa4558e9eacf2b81985"}
Nov 02 17:54:54.908471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:54.908447    5042 generic.go:356] "Generic (PLEG): container finished" podID="1a872d9c-5f74-4361-a14f-1914ca50de40" containerID="701c0737e9ddba1d5f8e28ddb08359134ead0694f1390d85c7f1e428f402b9ab" exitCode=0
Nov 02 17:54:54.908604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:54.908510    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht" event={"ID":"1a872d9c-5f74-4361-a14f-1914ca50de40","Type":"ContainerDied","Data":"701c0737e9ddba1d5f8e28ddb08359134ead0694f1390d85c7f1e428f402b9ab"}
Nov 02 17:54:56.227907 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.222104    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:56.228421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.228400    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:56.278500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.278468    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms\" (UniqueName: \"kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms\") pod \"f3b2f6bd-c213-409c-8885-517dd600d429\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") "
Nov 02 17:54:56.278705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.278523    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca\") pod \"1a872d9c-5f74-4361-a14f-1914ca50de40\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") "
Nov 02 17:54:56.278705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.278551    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca\") pod \"f3b2f6bd-c213-409c-8885-517dd600d429\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") "
Nov 02 17:54:56.278705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.278645    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg\" (UniqueName: \"kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg\") pod \"1a872d9c-5f74-4361-a14f-1914ca50de40\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") "
Nov 02 17:54:56.278705 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.278695    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca\") pod \"1a872d9c-5f74-4361-a14f-1914ca50de40\" (UID: \"1a872d9c-5f74-4361-a14f-1914ca50de40\") "
Nov 02 17:54:56.278939 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.278894    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca\") pod \"f3b2f6bd-c213-409c-8885-517dd600d429\" (UID: \"f3b2f6bd-c213-409c-8885-517dd600d429\") "
Nov 02 17:54:56.281154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.281122    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms" pod "f3b2f6bd-c213-409c-8885-517dd600d429" (UID: "f3b2f6bd-c213-409c-8885-517dd600d429"). InnerVolumeSpecName "kube-api-access-cfcms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:56.282165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.282140    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca" pod "f3b2f6bd-c213-409c-8885-517dd600d429" (UID: "f3b2f6bd-c213-409c-8885-517dd600d429"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:56.282546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.282522    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca" pod "1a872d9c-5f74-4361-a14f-1914ca50de40" (UID: "1a872d9c-5f74-4361-a14f-1914ca50de40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:56.293401 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.291431    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca" pod "f3b2f6bd-c213-409c-8885-517dd600d429" (UID: "f3b2f6bd-c213-409c-8885-517dd600d429"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:56.311278 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.309983    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca" pod "1a872d9c-5f74-4361-a14f-1914ca50de40" (UID: "1a872d9c-5f74-4361-a14f-1914ca50de40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:56.311278 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.310118    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg" pod "1a872d9c-5f74-4361-a14f-1914ca50de40" (UID: "1a872d9c-5f74-4361-a14f-1914ca50de40"). InnerVolumeSpecName "kube-api-access-kh5kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:56.346297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.346269    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380313    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380405    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380525    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380625    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380660    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380687    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380800    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.380953    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381010    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh\") pod \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\" (UID: \"bac100f1-0d8b-471c-9331-d2d5affd8dfd\") "
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381754    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh5kg\" (UniqueName: \"kubernetes.io/projected/1a872d9c-5f74-4361-a14f-1914ca50de40-kube-api-access-kh5kg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381778    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381795    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381808    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcms\" (UniqueName: \"kubernetes.io/projected/f3b2f6bd-c213-409c-8885-517dd600d429-kube-api-access-cfcms\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381822    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a872d9c-5f74-4361-a14f-1914ca50de40-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.385865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.381856    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2f6bd-c213-409c-8885-517dd600d429-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.394867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.394504    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:56.395492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.395278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:56.396961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.396357    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:56.397980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.397116    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:56.398338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.398300    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:56.409864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.401125    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:56.409864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.409537    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:56.409864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.409730    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "kube-api-access-tc6nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:56.412332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.412074    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy" pod "bac100f1-0d8b-471c-9331-d2d5affd8dfd" (UID: "bac100f1-0d8b-471c-9331-d2d5affd8dfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:56.483346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483252    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483289    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483304    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc6nh\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-kube-api-access-tc6nh\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483317    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483330    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483344    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483357    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483371    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bac100f1-0d8b-471c-9331-d2d5affd8dfd-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.483795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.483410    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bac100f1-0d8b-471c-9331-d2d5affd8dfd-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:56.927903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.927814    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"
Nov 02 17:54:56.928051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.927808    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv" event={"ID":"f3b2f6bd-c213-409c-8885-517dd600d429","Type":"ContainerDied","Data":"e4af85731cb0b0769b9ee2eeb460d82a9385d3f964443562a9e4853e7412a9b0"}
Nov 02 17:54:56.928051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.927980    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4af85731cb0b0769b9ee2eeb460d82a9385d3f964443562a9e4853e7412a9b0"
Nov 02 17:54:56.929947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.929922    5042 generic.go:356] "Generic (PLEG): container finished" podID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerID="270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81" exitCode=0
Nov 02 17:54:56.930085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.929984    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"
Nov 02 17:54:56.930085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.930012    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" event={"ID":"bac100f1-0d8b-471c-9331-d2d5affd8dfd","Type":"ContainerDied","Data":"270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81"}
Nov 02 17:54:56.930085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.930070    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" event={"ID":"bac100f1-0d8b-471c-9331-d2d5affd8dfd","Type":"ContainerDied","Data":"4f578aa033febcf3bba457af7cb6a2ba4b4904b8eb37088838a9c9e4b07c6668"}
Nov 02 17:54:56.930258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.930100    5042 scope.go:122] "RemoveContainer" containerID="270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81"
Nov 02 17:54:56.931974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.931944    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht" event={"ID":"1a872d9c-5f74-4361-a14f-1914ca50de40","Type":"ContainerDied","Data":"6c815f3cb46f540b11eaee4d89b7cc772509205fd08f45570f16fb82444dd95f"}
Nov 02 17:54:56.931974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.931968    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"
Nov 02 17:54:56.932140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.931997    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c815f3cb46f540b11eaee4d89b7cc772509205fd08f45570f16fb82444dd95f"
Nov 02 17:54:56.943424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.943404    5042 scope.go:122] "RemoveContainer" containerID="47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"
Nov 02 17:54:56.953771 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.953753    5042 scope.go:122] "RemoveContainer" containerID="270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81"
Nov 02 17:54:56.954090 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:56.954072    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81\": container with ID starting with 270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81 not found: ID does not exist" containerID="270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81"
Nov 02 17:54:56.954143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.954101    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81"} err="failed to get container status \"270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81\": rpc error: code = NotFound desc = could not find container \"270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81\": container with ID starting with 270a9d6ba1479dc4fdc280bb7afa356d2aebfbe73b8403ef995d0002a1dfca81 not found: ID does not exist"
Nov 02 17:54:56.954143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.954119    5042 scope.go:122] "RemoveContainer" containerID="47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"
Nov 02 17:54:56.954351 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:54:56.954332    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc\": container with ID starting with 47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc not found: ID does not exist" containerID="47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"
Nov 02 17:54:56.954391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.954357    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc"} err="failed to get container status \"47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc\": rpc error: code = NotFound desc = could not find container \"47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc\": container with ID starting with 47d0c60bbc09183d634fda3078665dfc5e763eb10925fa86f7cde5047e51e4fc not found: ID does not exist"
Nov 02 17:54:56.971352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.971297    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"]
Nov 02 17:54:56.975385 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:56.975356    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc"]
Nov 02 17:54:57.227897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:57.223146    5042 patch_prober.go:29] interesting pod/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:54:57.227897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:57.223231    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-e198eb-openshift-default-7554b6b489-b2wfc" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:57.546380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:57.546349    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" path="/var/lib/kubelet/pods/bac100f1-0d8b-471c-9331-d2d5affd8dfd/volumes"
Nov 02 17:54:57.942816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:57.942785    5042 generic.go:356] "Generic (PLEG): container finished" podID="621ff864-779d-4c0d-b079-8eff55464349" containerID="208f228e382f5dfb7e0ea2895c148fc019a7f1a7da3500b4eac7032a241b2643" exitCode=0
Nov 02 17:54:57.942991 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:57.942865    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" event={"ID":"621ff864-779d-4c0d-b079-8eff55464349","Type":"ContainerDied","Data":"208f228e382f5dfb7e0ea2895c148fc019a7f1a7da3500b4eac7032a241b2643"}
Nov 02 17:54:58.223334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.223302    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:54:58.331788 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.331752    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.331806    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.331873    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.331924    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.331979    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332061    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332067    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332110    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332145    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332196    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332270 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332229    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy\") pod \"621ff864-779d-4c0d-b079-8eff55464349\" (UID: \"621ff864-779d-4c0d-b079-8eff55464349\") "
Nov 02 17:54:58.332864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332306    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:58.332864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332324    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:58.332864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.332599    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:58.333203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.333085    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.333203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.333107    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.333203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.333122    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/621ff864-779d-4c0d-b079-8eff55464349-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.333203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.333137    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.333203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.333150    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.346095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.334743    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:58.346095 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.341371    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "kube-api-access-tgcb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:58.346723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.346695    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:58.346723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.346646    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token" pod "621ff864-779d-4c0d-b079-8eff55464349" (UID: "621ff864-779d-4c0d-b079-8eff55464349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:58.385123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385089    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"]
Nov 02 17:54:58.385712 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385698    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy"
Nov 02 17:54:58.385757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385714    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy"
Nov 02 17:54:58.385757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385736    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f3b2f6bd-c213-409c-8885-517dd600d429" containerName="revoke-keys"
Nov 02 17:54:58.385757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385741    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b2f6bd-c213-409c-8885-517dd600d429" containerName="revoke-keys"
Nov 02 17:54:58.385894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385760    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy"
Nov 02 17:54:58.385894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385765    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy"
Nov 02 17:54:58.385894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385783    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1a872d9c-5f74-4361-a14f-1914ca50de40" containerName="revoke-keys"
Nov 02 17:54:58.385894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385788    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a872d9c-5f74-4361-a14f-1914ca50de40" containerName="revoke-keys"
Nov 02 17:54:58.385894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385799    5042 cpu_manager.go:414] "containerMap: removing container" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy"
Nov 02 17:54:58.385894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385804    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy"
Nov 02 17:54:58.386071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385907    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b2f6bd-c213-409c-8885-517dd600d429" containerName="revoke-keys"
Nov 02 17:54:58.386071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385919    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy"
Nov 02 17:54:58.386071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385928    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a872d9c-5f74-4361-a14f-1914ca50de40" containerName="revoke-keys"
Nov 02 17:54:58.386071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385934    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="621ff864-779d-4c0d-b079-8eff55464349" containerName="istio-proxy"
Nov 02 17:54:58.386071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.385941    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac100f1-0d8b-471c-9331-d2d5affd8dfd" containerName="istio-proxy"
Nov 02 17:54:58.389645 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.389625    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.396508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.396478    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:54:58.405539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.405511    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"]
Nov 02 17:54:58.405768 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.405752    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:54:58.405938 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.405874    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:54:58.434462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.434434    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/621ff864-779d-4c0d-b079-8eff55464349-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.434462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.434463    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgcb8\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-kube-api-access-tgcb8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.434626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.434472    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/621ff864-779d-4c0d-b079-8eff55464349-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.434626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.434481    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/621ff864-779d-4c0d-b079-8eff55464349-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:54:58.496761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.496673    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:58.535464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.535432    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.535667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.535599    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94xc\" (UniqueName: \"kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.535667 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.535646    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.637137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.637099    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94xc\" (UniqueName: \"kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.637325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.637162    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.637426 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.637392    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.637880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.637826    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.639882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.639865    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.646236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.646215    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94xc\" (UniqueName: \"kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc\") pod \"maas-api-revoke-sub-b70b4c74-qczsz\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.709129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.709093    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:54:58.956760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.956736    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"
Nov 02 17:54:58.956760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.956749    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68" event={"ID":"621ff864-779d-4c0d-b079-8eff55464349","Type":"ContainerDied","Data":"1b5ef4bce26fba7c9ab0a4073e9078e51a5b12cfbea4dfeb5fd087c058966f1d"}
Nov 02 17:54:58.957015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.956812    5042 scope.go:122] "RemoveContainer" containerID="208f228e382f5dfb7e0ea2895c148fc019a7f1a7da3500b4eac7032a241b2643"
Nov 02 17:54:58.990137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.990102    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"]
Nov 02 17:54:58.994275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:58.994252    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-sgx68"]
Nov 02 17:54:59.002440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:59.002410    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"]
Nov 02 17:54:59.003272 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:54:59.003250    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713f720b_b35a_4079_8f66_0faaba6c5265.slice/crio-d1a20ce1167f3deb71ae3675e9288a7dc8c87c9d0bb3f820c49dc708f3549d7f WatchSource:0}: Error finding container d1a20ce1167f3deb71ae3675e9288a7dc8c87c9d0bb3f820c49dc708f3549d7f: Status 404 returned error can't find the container with id d1a20ce1167f3deb71ae3675e9288a7dc8c87c9d0bb3f820c49dc708f3549d7f
Nov 02 17:54:59.546024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:59.545946    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621ff864-779d-4c0d-b079-8eff55464349" path="/var/lib/kubelet/pods/621ff864-779d-4c0d-b079-8eff55464349/volumes"
Nov 02 17:54:59.967255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:59.967223    5042 generic.go:356] "Generic (PLEG): container finished" podID="713f720b-b35a-4079-8f66-0faaba6c5265" containerID="c6c0e1e0d7b7ede5f23f331c24784f78f13e53dcc6dca988736f6b2dad4495b1" exitCode=0
Nov 02 17:54:59.967428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:59.967304    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz" event={"ID":"713f720b-b35a-4079-8f66-0faaba6c5265","Type":"ContainerDied","Data":"c6c0e1e0d7b7ede5f23f331c24784f78f13e53dcc6dca988736f6b2dad4495b1"}
Nov 02 17:54:59.967428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:54:59.967353    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz" event={"ID":"713f720b-b35a-4079-8f66-0faaba6c5265","Type":"ContainerStarted","Data":"d1a20ce1167f3deb71ae3675e9288a7dc8c87c9d0bb3f820c49dc708f3549d7f"}
Nov 02 17:55:01.041703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.041668    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"]
Nov 02 17:55:01.056321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.056292    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"]
Nov 02 17:55:01.187803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.187776    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:55:01.283018 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.282926    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca\") pod \"713f720b-b35a-4079-8f66-0faaba6c5265\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") "
Nov 02 17:55:01.283198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.283145    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca\") pod \"713f720b-b35a-4079-8f66-0faaba6c5265\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") "
Nov 02 17:55:01.283264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.283235    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc\" (UniqueName: \"kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc\") pod \"713f720b-b35a-4079-8f66-0faaba6c5265\" (UID: \"713f720b-b35a-4079-8f66-0faaba6c5265\") "
Nov 02 17:55:01.283542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.283516    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca" pod "713f720b-b35a-4079-8f66-0faaba6c5265" (UID: "713f720b-b35a-4079-8f66-0faaba6c5265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:01.283874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.283858    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:01.285881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.285573    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca" pod "713f720b-b35a-4079-8f66-0faaba6c5265" (UID: "713f720b-b35a-4079-8f66-0faaba6c5265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:01.286017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.285908    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc" pod "713f720b-b35a-4079-8f66-0faaba6c5265" (UID: "713f720b-b35a-4079-8f66-0faaba6c5265"). InnerVolumeSpecName "kube-api-access-q94xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:01.427866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.427442    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94xc\" (UniqueName: \"kubernetes.io/projected/713f720b-b35a-4079-8f66-0faaba6c5265-kube-api-access-q94xc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:01.427866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.427479    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713f720b-b35a-4079-8f66-0faaba6c5265-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:01.984745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.984718    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"
Nov 02 17:55:01.984945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.984753    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz" event={"ID":"713f720b-b35a-4079-8f66-0faaba6c5265","Type":"ContainerDied","Data":"d1a20ce1167f3deb71ae3675e9288a7dc8c87c9d0bb3f820c49dc708f3549d7f"}
Nov 02 17:55:01.984945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:01.984804    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1a20ce1167f3deb71ae3675e9288a7dc8c87c9d0bb3f820c49dc708f3549d7f"
Nov 02 17:55:03.688598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.688559    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"]
Nov 02 17:55:03.689482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.689459    5042 cpu_manager.go:414] "containerMap: removing container" podUID="713f720b-b35a-4079-8f66-0faaba6c5265" containerName="revoke-keys"
Nov 02 17:55:03.689528 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.689486    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713f720b-b35a-4079-8f66-0faaba6c5265" containerName="revoke-keys"
Nov 02 17:55:03.689743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.689728    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="713f720b-b35a-4079-8f66-0faaba6c5265" containerName="revoke-keys"
Nov 02 17:55:03.694282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.694261    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.698640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.698615    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:55:03.698916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.698832    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:55:03.699656 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.699613    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:55:03.699656 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.699620    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:55:03.701143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.701114    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"]
Nov 02 17:55:03.768102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.768058    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.768360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.768142    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.768465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.768446    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfzj\" (UniqueName: \"kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.869569 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.869534    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfzj\" (UniqueName: \"kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.869723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.869577    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.869723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.869614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.870333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.870313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.872185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.872165    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:03.878761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:03.878730    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfzj\" (UniqueName: \"kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj\") pod \"maas-api-revoke-sub-e6bfaf27-49r5g\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:04.017268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:04.017185    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:04.305017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:04.304987    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"]
Nov 02 17:55:04.306823 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:04.306793    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbd3ead_5ba3_4f0b_afb7_8bb637979dc6.slice/crio-26e300569d1018a77e30ecd9dd9cfb63ea2ae684a2de55eeedc2d38bd8cbc343 WatchSource:0}: Error finding container 26e300569d1018a77e30ecd9dd9cfb63ea2ae684a2de55eeedc2d38bd8cbc343: Status 404 returned error can't find the container with id 26e300569d1018a77e30ecd9dd9cfb63ea2ae684a2de55eeedc2d38bd8cbc343
Nov 02 17:55:05.013574 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.013523    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g" event={"ID":"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6","Type":"ContainerStarted","Data":"3f1f05d38425fbd72972ec933590e9aefdbedf3cc51b8df1a2d93897eaba32db"}
Nov 02 17:55:05.014201 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.013632    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g" event={"ID":"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6","Type":"ContainerStarted","Data":"26e300569d1018a77e30ecd9dd9cfb63ea2ae684a2de55eeedc2d38bd8cbc343"}
Nov 02 17:55:05.016203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.016179    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"]
Nov 02 17:55:05.020176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.020158    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.029390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.029233    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"]
Nov 02 17:55:05.038205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.038149    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g" podStartSLOduration=2.038131033 podStartE2EDuration="2.038131033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:05.031792113 +0000 UTC m=+1317.611201511" watchObservedRunningTime="2026-11-02 17:55:05.038131033 +0000 UTC m=+1317.617540419"
Nov 02 17:55:05.096763 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.096726    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.097124 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.097092    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.097248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.097149    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5m9w\" (UniqueName: \"kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.199357 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.199321    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.199856 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.199816    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.200014 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.199997    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5m9w\" (UniqueName: \"kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.201687 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.201665    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.205135 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.205110    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.209635 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.209606    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5m9w\" (UniqueName: \"kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w\") pod \"maas-api-revoke-sub-c9a3b56a-cnclg\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.345156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.345056    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:05.634408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:05.634377    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"]
Nov 02 17:55:05.635289 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:05.635258    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0948114_a0a8_406c_b596_eb74f7309623.slice/crio-2066f752214a538ff1114ae982b616e3c59f5867b9d193e7bb4ef424bf31dc53 WatchSource:0}: Error finding container 2066f752214a538ff1114ae982b616e3c59f5867b9d193e7bb4ef424bf31dc53: Status 404 returned error can't find the container with id 2066f752214a538ff1114ae982b616e3c59f5867b9d193e7bb4ef424bf31dc53
Nov 02 17:55:06.025681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.025635    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg" event={"ID":"d0948114-a0a8-406c-b596-eb74f7309623","Type":"ContainerStarted","Data":"6e2baee3302593c9bd285c4a487da3a6b9a3e30db8b24a80ef52b6896c1b9d15"}
Nov 02 17:55:06.026224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.025698    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg" event={"ID":"d0948114-a0a8-406c-b596-eb74f7309623","Type":"ContainerStarted","Data":"2066f752214a538ff1114ae982b616e3c59f5867b9d193e7bb4ef424bf31dc53"}
Nov 02 17:55:06.027438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.027414    5042 generic.go:356] "Generic (PLEG): container finished" podID="5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" containerID="3f1f05d38425fbd72972ec933590e9aefdbedf3cc51b8df1a2d93897eaba32db" exitCode=0
Nov 02 17:55:06.027573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.027490    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g" event={"ID":"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6","Type":"ContainerDied","Data":"3f1f05d38425fbd72972ec933590e9aefdbedf3cc51b8df1a2d93897eaba32db"}
Nov 02 17:55:06.043640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.043576    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm" podUID="3830c9d3-c656-4b15-9757-474a158450d8" containerName="payload-processing" containerID="cri-o://b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440" gracePeriod=25
Nov 02 17:55:06.052238 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.052187    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg" podStartSLOduration=2.052172286 podStartE2EDuration="2.052172286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:06.047701512 +0000 UTC m=+1318.627110898" watchObservedRunningTime="2026-11-02 17:55:06.052172286 +0000 UTC m=+1318.631581672"
Nov 02 17:55:06.056681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.056641    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx" podUID="593f187e-2b2d-4e4d-9762-332be46734c1" containerName="payload-pre-processing" containerID="cri-o://4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8" gracePeriod=25
Nov 02 17:55:06.373473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.373441    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:55:06.386624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.386598    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:55:06.423784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.422224    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t\" (UniqueName: \"kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t\") pod \"3830c9d3-c656-4b15-9757-474a158450d8\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") "
Nov 02 17:55:06.423784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.422498    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume\") pod \"3830c9d3-c656-4b15-9757-474a158450d8\" (UID: \"3830c9d3-c656-4b15-9757-474a158450d8\") "
Nov 02 17:55:06.423784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.422896    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume" pod "3830c9d3-c656-4b15-9757-474a158450d8" (UID: "3830c9d3-c656-4b15-9757-474a158450d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:06.423784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.423163    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3830c9d3-c656-4b15-9757-474a158450d8-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:06.425739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.425716    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t" pod "3830c9d3-c656-4b15-9757-474a158450d8" (UID: "3830c9d3-c656-4b15-9757-474a158450d8"). InnerVolumeSpecName "kube-api-access-bfs7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:06.524525 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.524496    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64\" (UniqueName: \"kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64\") pod \"593f187e-2b2d-4e4d-9762-332be46734c1\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") "
Nov 02 17:55:06.524724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.524595    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume\") pod \"593f187e-2b2d-4e4d-9762-332be46734c1\" (UID: \"593f187e-2b2d-4e4d-9762-332be46734c1\") "
Nov 02 17:55:06.524998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.524974    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume" pod "593f187e-2b2d-4e4d-9762-332be46734c1" (UID: "593f187e-2b2d-4e4d-9762-332be46734c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:06.525214 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.525198    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfs7t\" (UniqueName: \"kubernetes.io/projected/3830c9d3-c656-4b15-9757-474a158450d8-kube-api-access-bfs7t\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:06.525277 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.525216    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/593f187e-2b2d-4e4d-9762-332be46734c1-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:06.526800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.526780    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64" pod "593f187e-2b2d-4e4d-9762-332be46734c1" (UID: "593f187e-2b2d-4e4d-9762-332be46734c1"). InnerVolumeSpecName "kube-api-access-z6f64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:06.626824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:06.626793    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6f64\" (UniqueName: \"kubernetes.io/projected/593f187e-2b2d-4e4d-9762-332be46734c1-kube-api-access-z6f64\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:07.036822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.036791    5042 generic.go:356] "Generic (PLEG): container finished" podID="593f187e-2b2d-4e4d-9762-332be46734c1" containerID="4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8" exitCode=0
Nov 02 17:55:07.037302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.036878    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"
Nov 02 17:55:07.037302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.036874    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx" event={"ID":"593f187e-2b2d-4e4d-9762-332be46734c1","Type":"ContainerDied","Data":"4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8"}
Nov 02 17:55:07.037302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.036936    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx" event={"ID":"593f187e-2b2d-4e4d-9762-332be46734c1","Type":"ContainerDied","Data":"b9cd3fef89fcc4bb238bb7d4ce092d09f050b66a4916b843181dad2372530959"}
Nov 02 17:55:07.037302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.036968    5042 scope.go:122] "RemoveContainer" containerID="4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8"
Nov 02 17:55:07.038473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.038449    5042 generic.go:356] "Generic (PLEG): container finished" podID="3830c9d3-c656-4b15-9757-474a158450d8" containerID="b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440" exitCode=0
Nov 02 17:55:07.038596 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.038498    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"
Nov 02 17:55:07.038596 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.038552    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm" event={"ID":"3830c9d3-c656-4b15-9757-474a158450d8","Type":"ContainerDied","Data":"b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440"}
Nov 02 17:55:07.038596 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.038591    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm" event={"ID":"3830c9d3-c656-4b15-9757-474a158450d8","Type":"ContainerDied","Data":"db74aa6656017187e0bd6cccb3f40ad6fada5e7d940a2b978491720c3d1fb338"}
Nov 02 17:55:07.040504 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.040478    5042 generic.go:356] "Generic (PLEG): container finished" podID="d0948114-a0a8-406c-b596-eb74f7309623" containerID="6e2baee3302593c9bd285c4a487da3a6b9a3e30db8b24a80ef52b6896c1b9d15" exitCode=0
Nov 02 17:55:07.040630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.040565    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg" event={"ID":"d0948114-a0a8-406c-b596-eb74f7309623","Type":"ContainerDied","Data":"6e2baee3302593c9bd285c4a487da3a6b9a3e30db8b24a80ef52b6896c1b9d15"}
Nov 02 17:55:07.051096 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.051075    5042 scope.go:122] "RemoveContainer" containerID="4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8"
Nov 02 17:55:07.051406 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:55:07.051389    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8\": container with ID starting with 4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8 not found: ID does not exist" containerID="4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8"
Nov 02 17:55:07.051462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.051414    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8"} err="failed to get container status \"4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8\": rpc error: code = NotFound desc = could not find container \"4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8\": container with ID starting with 4415f6cfe65b0e4efae6cdcf27d4157e0f942755c1992b49010bcc08cc6846d8 not found: ID does not exist"
Nov 02 17:55:07.051462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.051430    5042 scope.go:122] "RemoveContainer" containerID="b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440"
Nov 02 17:55:07.065349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.065322    5042 scope.go:122] "RemoveContainer" containerID="b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440"
Nov 02 17:55:07.065705 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:55:07.065681    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440\": container with ID starting with b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440 not found: ID does not exist" containerID="b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440"
Nov 02 17:55:07.065794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.065714    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440"} err="failed to get container status \"b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440\": rpc error: code = NotFound desc = could not find container \"b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440\": container with ID starting with b743857ac10dbc164b6505ec047a4d9d4ebad1032e035021d60c06084381f440 not found: ID does not exist"
Nov 02 17:55:07.084503 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.084472    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"]
Nov 02 17:55:07.088552 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.088524    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dfd356-84b4fdcbdc-2nldm"]
Nov 02 17:55:07.103118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.103085    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"]
Nov 02 17:55:07.114482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.109916    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-dfd356-7f4db9cc6c-6qjtx"]
Nov 02 17:55:07.255179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.255154    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:07.340637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.340602    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca\") pod \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") "
Nov 02 17:55:07.340877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.340707    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj\" (UniqueName: \"kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj\") pod \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") "
Nov 02 17:55:07.340961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.340900    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca\") pod \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\" (UID: \"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6\") "
Nov 02 17:55:07.341073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.341045    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca" pod "5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" (UID: "5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:07.341410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.341391    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:07.343011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.342984    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj" pod "5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" (UID: "5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6"). InnerVolumeSpecName "kube-api-access-mtfzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:07.343112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.343043    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca" pod "5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" (UID: "5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:07.443164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.443122    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfzj\" (UniqueName: \"kubernetes.io/projected/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-kube-api-access-mtfzj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:07.443164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.443153    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:07.594123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.590280    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3830c9d3-c656-4b15-9757-474a158450d8" path="/var/lib/kubelet/pods/3830c9d3-c656-4b15-9757-474a158450d8/volumes"
Nov 02 17:55:07.594123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:07.590919    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="593f187e-2b2d-4e4d-9762-332be46734c1" path="/var/lib/kubelet/pods/593f187e-2b2d-4e4d-9762-332be46734c1/volumes"
Nov 02 17:55:08.049488 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.049443    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g" event={"ID":"5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6","Type":"ContainerDied","Data":"26e300569d1018a77e30ecd9dd9cfb63ea2ae684a2de55eeedc2d38bd8cbc343"}
Nov 02 17:55:08.049985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.049532    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26e300569d1018a77e30ecd9dd9cfb63ea2ae684a2de55eeedc2d38bd8cbc343"
Nov 02 17:55:08.049985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.049465    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"
Nov 02 17:55:08.222320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.222293    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:08.329458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.324050    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca\") pod \"d0948114-a0a8-406c-b596-eb74f7309623\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") "
Nov 02 17:55:08.329458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.324301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w\" (UniqueName: \"kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w\") pod \"d0948114-a0a8-406c-b596-eb74f7309623\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") "
Nov 02 17:55:08.329458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.324378    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca\") pod \"d0948114-a0a8-406c-b596-eb74f7309623\" (UID: \"d0948114-a0a8-406c-b596-eb74f7309623\") "
Nov 02 17:55:08.329458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.324575    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca" pod "d0948114-a0a8-406c-b596-eb74f7309623" (UID: "d0948114-a0a8-406c-b596-eb74f7309623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:08.329458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.325260    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:08.330587 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.330550    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w" pod "d0948114-a0a8-406c-b596-eb74f7309623" (UID: "d0948114-a0a8-406c-b596-eb74f7309623"). InnerVolumeSpecName "kube-api-access-j5m9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:08.335520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.335403    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca" pod "d0948114-a0a8-406c-b596-eb74f7309623" (UID: "d0948114-a0a8-406c-b596-eb74f7309623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:08.427062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.427029    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5m9w\" (UniqueName: \"kubernetes.io/projected/d0948114-a0a8-406c-b596-eb74f7309623-kube-api-access-j5m9w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:08.427062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:08.427059    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0948114-a0a8-406c-b596-eb74f7309623-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:09.079915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:09.079873    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"
Nov 02 17:55:09.080438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:09.080012    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg" event={"ID":"d0948114-a0a8-406c-b596-eb74f7309623","Type":"ContainerDied","Data":"2066f752214a538ff1114ae982b616e3c59f5867b9d193e7bb4ef424bf31dc53"}
Nov 02 17:55:09.080438 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:09.080071    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2066f752214a538ff1114ae982b616e3c59f5867b9d193e7bb4ef424bf31dc53"
Nov 02 17:55:10.091220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:10.091173    5042 scope.go:122] "RemoveContainer" containerID="72976f457df52ea8681039ab008de3d47cc5b24cf011b17f8db8453218206d77"
Nov 02 17:55:10.223026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:10.222880    5042 scope.go:122] "RemoveContainer" containerID="3223110529f067a6e7edeac575521329fb9b21957735a3dfbe7508945baeec5b"
Nov 02 17:55:10.239530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:10.239504    5042 scope.go:122] "RemoveContainer" containerID="7924bfe2f3ef74a92f35b551ed10b8766f1460182b8912daa1acaf7ddecf870e"
Nov 02 17:55:11.513879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:11.510942    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:11.561877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:11.559977    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:11.848926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:11.845734    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:12.511866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:12.511459    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:12.562880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:12.512501    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:55:13.167016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:13.166985    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:55:17.619182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619145    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"]
Nov 02 17:55:17.619727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619713    5042 cpu_manager.go:414] "containerMap: removing container" podUID="593f187e-2b2d-4e4d-9762-332be46734c1" containerName="payload-pre-processing"
Nov 02 17:55:17.619777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619728    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="593f187e-2b2d-4e4d-9762-332be46734c1" containerName="payload-pre-processing"
Nov 02 17:55:17.619777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619750    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3830c9d3-c656-4b15-9757-474a158450d8" containerName="payload-processing"
Nov 02 17:55:17.619777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619756    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3830c9d3-c656-4b15-9757-474a158450d8" containerName="payload-processing"
Nov 02 17:55:17.619777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619772    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d0948114-a0a8-406c-b596-eb74f7309623" containerName="revoke-keys"
Nov 02 17:55:17.619777 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619777    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0948114-a0a8-406c-b596-eb74f7309623" containerName="revoke-keys"
Nov 02 17:55:17.619949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619796    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" containerName="revoke-keys"
Nov 02 17:55:17.619949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619801    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" containerName="revoke-keys"
Nov 02 17:55:17.619949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619919    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3830c9d3-c656-4b15-9757-474a158450d8" containerName="payload-processing"
Nov 02 17:55:17.619949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619930    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0948114-a0a8-406c-b596-eb74f7309623" containerName="revoke-keys"
Nov 02 17:55:17.619949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619939    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" containerName="revoke-keys"
Nov 02 17:55:17.619949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.619949    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="593f187e-2b2d-4e4d-9762-332be46734c1" containerName="payload-pre-processing"
Nov 02 17:55:17.625297 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.625277    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.632430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.632406    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:55:17.632600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.632450    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:55:17.632600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.632570    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:55:17.632714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.632601    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:55:17.633154 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.633126    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"]
Nov 02 17:55:17.792002 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.791966    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv78s\" (UniqueName: \"kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.792187 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.792168    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.792325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.792297    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.893760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.893674    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.893760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.893749    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.894023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.893872    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv78s\" (UniqueName: \"kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.894534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.894461    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.896298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.896277    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.905769 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.905741    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv78s\" (UniqueName: \"kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s\") pod \"maas-api-revoke-sub-314c8721-gpk8x\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:17.942919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:17.942875    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:18.234984 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:18.234951    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"]
Nov 02 17:55:18.235968 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:18.235940    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24432ad9_44d9_4ae0_95bb_6515a2d97b9e.slice/crio-fe9d7a57140197977cfebdb37304b3a8f5c024b46d9124e98e97ec05a99789f2 WatchSource:0}: Error finding container fe9d7a57140197977cfebdb37304b3a8f5c024b46d9124e98e97ec05a99789f2: Status 404 returned error can't find the container with id fe9d7a57140197977cfebdb37304b3a8f5c024b46d9124e98e97ec05a99789f2
Nov 02 17:55:18.292210 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:18.292167    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x" event={"ID":"24432ad9-44d9-4ae0-95bb-6515a2d97b9e","Type":"ContainerStarted","Data":"fe9d7a57140197977cfebdb37304b3a8f5c024b46d9124e98e97ec05a99789f2"}
Nov 02 17:55:18.541910 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:55:18.541869    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:55:19.305477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:19.305440    5042 generic.go:356] "Generic (PLEG): container finished" podID="24432ad9-44d9-4ae0-95bb-6515a2d97b9e" containerID="a41c1bcad832411c312e06da1fe5e05e9ee066a6016e9edc40fc5fe5bf7b4273" exitCode=0
Nov 02 17:55:19.306039 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:19.305537    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x" event={"ID":"24432ad9-44d9-4ae0-95bb-6515a2d97b9e","Type":"ContainerDied","Data":"a41c1bcad832411c312e06da1fe5e05e9ee066a6016e9edc40fc5fe5bf7b4273"}
Nov 02 17:55:20.566607 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.566536    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:20.643585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.643547    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca\") pod \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") "
Nov 02 17:55:20.643805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.643718    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca\") pod \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") "
Nov 02 17:55:20.643805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.643792    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s\" (UniqueName: \"kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s\") pod \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\" (UID: \"24432ad9-44d9-4ae0-95bb-6515a2d97b9e\") "
Nov 02 17:55:20.644072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.644042    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca" pod "24432ad9-44d9-4ae0-95bb-6515a2d97b9e" (UID: "24432ad9-44d9-4ae0-95bb-6515a2d97b9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:20.644549 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.644519    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:20.646259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.646234    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca" pod "24432ad9-44d9-4ae0-95bb-6515a2d97b9e" (UID: "24432ad9-44d9-4ae0-95bb-6515a2d97b9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:20.646259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.646241    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s" pod "24432ad9-44d9-4ae0-95bb-6515a2d97b9e" (UID: "24432ad9-44d9-4ae0-95bb-6515a2d97b9e"). InnerVolumeSpecName "kube-api-access-zv78s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:20.745977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.745936    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:20.745977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.745973    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv78s\" (UniqueName: \"kubernetes.io/projected/24432ad9-44d9-4ae0-95bb-6515a2d97b9e-kube-api-access-zv78s\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:20.897337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.897252    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"]
Nov 02 17:55:20.897900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.897880    5042 cpu_manager.go:414] "containerMap: removing container" podUID="24432ad9-44d9-4ae0-95bb-6515a2d97b9e" containerName="revoke-keys"
Nov 02 17:55:20.897992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.897901    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24432ad9-44d9-4ae0-95bb-6515a2d97b9e" containerName="revoke-keys"
Nov 02 17:55:20.898119 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.898098    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="24432ad9-44d9-4ae0-95bb-6515a2d97b9e" containerName="revoke-keys"
Nov 02 17:55:20.902564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.902544    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:20.911497 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.911469    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"]
Nov 02 17:55:20.948515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.948477    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:20.948719 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.948544    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:20.948910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:20.948887    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz9d8\" (UniqueName: \"kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.050075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.050041    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9d8\" (UniqueName: \"kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.050075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.050079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.050313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.050105    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.050683 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.050656    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.052558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.052534    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.066437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.066410    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9d8\" (UniqueName: \"kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8\") pod \"maas-api-revoke-sub-33b05595-556xm\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.225300 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.225265    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:21.362883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.358710    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x" event={"ID":"24432ad9-44d9-4ae0-95bb-6515a2d97b9e","Type":"ContainerDied","Data":"fe9d7a57140197977cfebdb37304b3a8f5c024b46d9124e98e97ec05a99789f2"}
Nov 02 17:55:21.362883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.358765    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe9d7a57140197977cfebdb37304b3a8f5c024b46d9124e98e97ec05a99789f2"
Nov 02 17:55:21.362883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.358822    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"
Nov 02 17:55:21.513226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:21.513200    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"]
Nov 02 17:55:21.514624 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:21.514594    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c7a58d_81cd_4b25_a3fa_457afef7fe3d.slice/crio-36140992b7e4feeaa8218eea4f4a8db686fe611cb801b4c51502a40f3046de4a WatchSource:0}: Error finding container 36140992b7e4feeaa8218eea4f4a8db686fe611cb801b4c51502a40f3046de4a: Status 404 returned error can't find the container with id 36140992b7e4feeaa8218eea4f4a8db686fe611cb801b4c51502a40f3046de4a
Nov 02 17:55:22.367500 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:22.367458    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm" event={"ID":"94c7a58d-81cd-4b25-a3fa-457afef7fe3d","Type":"ContainerStarted","Data":"694c9254437c9f842255ea101b0b184b503e837a56accc14d4dff35d9063e91e"}
Nov 02 17:55:22.367973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:22.367515    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm" event={"ID":"94c7a58d-81cd-4b25-a3fa-457afef7fe3d","Type":"ContainerStarted","Data":"36140992b7e4feeaa8218eea4f4a8db686fe611cb801b4c51502a40f3046de4a"}
Nov 02 17:55:22.390067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:22.390004    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm" podStartSLOduration=2.389984498 podStartE2EDuration="2.389984498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:22.388875271 +0000 UTC m=+1334.968284658" watchObservedRunningTime="2026-11-02 17:55:22.389984498 +0000 UTC m=+1334.969393884"
Nov 02 17:55:23.376803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:23.376769    5042 generic.go:356] "Generic (PLEG): container finished" podID="94c7a58d-81cd-4b25-a3fa-457afef7fe3d" containerID="694c9254437c9f842255ea101b0b184b503e837a56accc14d4dff35d9063e91e" exitCode=0
Nov 02 17:55:23.376803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:23.376808    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm" event={"ID":"94c7a58d-81cd-4b25-a3fa-457afef7fe3d","Type":"ContainerDied","Data":"694c9254437c9f842255ea101b0b184b503e837a56accc14d4dff35d9063e91e"}
Nov 02 17:55:24.586041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.586005    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:24.719959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.719919    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca\") pod \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") "
Nov 02 17:55:24.720169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.720077    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8\" (UniqueName: \"kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8\") pod \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") "
Nov 02 17:55:24.720169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.720107    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca\") pod \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\" (UID: \"94c7a58d-81cd-4b25-a3fa-457afef7fe3d\") "
Nov 02 17:55:24.720408 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.720378    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca" pod "94c7a58d-81cd-4b25-a3fa-457afef7fe3d" (UID: "94c7a58d-81cd-4b25-a3fa-457afef7fe3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:24.720826 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.720799    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:24.723168 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.723133    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca" pod "94c7a58d-81cd-4b25-a3fa-457afef7fe3d" (UID: "94c7a58d-81cd-4b25-a3fa-457afef7fe3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:24.723429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.723408    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8" pod "94c7a58d-81cd-4b25-a3fa-457afef7fe3d" (UID: "94c7a58d-81cd-4b25-a3fa-457afef7fe3d"). InnerVolumeSpecName "kube-api-access-vz9d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:24.822594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.822552    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz9d8\" (UniqueName: \"kubernetes.io/projected/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-kube-api-access-vz9d8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:24.822874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:24.822833    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94c7a58d-81cd-4b25-a3fa-457afef7fe3d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:25.089825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.089737    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"]
Nov 02 17:55:25.090668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.090645    5042 cpu_manager.go:414] "containerMap: removing container" podUID="94c7a58d-81cd-4b25-a3fa-457afef7fe3d" containerName="revoke-keys"
Nov 02 17:55:25.090715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.090674    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c7a58d-81cd-4b25-a3fa-457afef7fe3d" containerName="revoke-keys"
Nov 02 17:55:25.090922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.090904    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c7a58d-81cd-4b25-a3fa-457afef7fe3d" containerName="revoke-keys"
Nov 02 17:55:25.095224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.095205    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.102824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.102798    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"]
Nov 02 17:55:25.233690 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.233640    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.233954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.233707    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcgmj\" (UniqueName: \"kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.233954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.233893    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.335499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.335462    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.335706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.335507    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcgmj\" (UniqueName: \"kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.335706 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.335557    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.336222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.336198    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.338349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.338321    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.345695 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.345636    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcgmj\" (UniqueName: \"kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj\") pod \"maas-api-revoke-sub-e4db47eb-znxsw\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.423919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.423817    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:25.426956 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.426900    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm" event={"ID":"94c7a58d-81cd-4b25-a3fa-457afef7fe3d","Type":"ContainerDied","Data":"36140992b7e4feeaa8218eea4f4a8db686fe611cb801b4c51502a40f3046de4a"}
Nov 02 17:55:25.427191 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.427176    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36140992b7e4feeaa8218eea4f4a8db686fe611cb801b4c51502a40f3046de4a"
Nov 02 17:55:25.427318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.426972    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"
Nov 02 17:55:25.930978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:25.927792    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"]
Nov 02 17:55:25.940725 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:25.940687    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ea8853_9e7c_487a_ac09_275f3a589b54.slice/crio-e05d45f141e459614752f9df1f84aa14bd21b6a731fbce5f42ed8ba53f31910e WatchSource:0}: Error finding container e05d45f141e459614752f9df1f84aa14bd21b6a731fbce5f42ed8ba53f31910e: Status 404 returned error can't find the container with id e05d45f141e459614752f9df1f84aa14bd21b6a731fbce5f42ed8ba53f31910e
Nov 02 17:55:26.529951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:26.529550    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw" event={"ID":"05ea8853-9e7c-487a-ac09-275f3a589b54","Type":"ContainerStarted","Data":"e05d45f141e459614752f9df1f84aa14bd21b6a731fbce5f42ed8ba53f31910e"}
Nov 02 17:55:27.326588 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:55:27.326546    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ea8853_9e7c_487a_ac09_275f3a589b54.slice/crio-cc85c46d02598dc044b6b001ce182ef2acd860f5e2409de9c23ac6fff89d021b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 17:55:27.509868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:27.507171    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:27.598537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:27.571952    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:55:27.843861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:27.843116    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:27.906056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:27.905962    5042 generic.go:356] "Generic (PLEG): container finished" podID="05ea8853-9e7c-487a-ac09-275f3a589b54" containerID="cc85c46d02598dc044b6b001ce182ef2acd860f5e2409de9c23ac6fff89d021b" exitCode=0
Nov 02 17:55:27.906289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:27.906271    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw" event={"ID":"05ea8853-9e7c-487a-ac09-275f3a589b54","Type":"ContainerDied","Data":"cc85c46d02598dc044b6b001ce182ef2acd860f5e2409de9c23ac6fff89d021b"}
Nov 02 17:55:27.928125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:27.928082    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:29.094855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.094812    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:29.122085 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.122046    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj\" (UniqueName: \"kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj\") pod \"05ea8853-9e7c-487a-ac09-275f3a589b54\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") "
Nov 02 17:55:29.122357 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.122201    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca\") pod \"05ea8853-9e7c-487a-ac09-275f3a589b54\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") "
Nov 02 17:55:29.124347 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.124294    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca" pod "05ea8853-9e7c-487a-ac09-275f3a589b54" (UID: "05ea8853-9e7c-487a-ac09-275f3a589b54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:29.127114 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.126908    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca\") pod \"05ea8853-9e7c-487a-ac09-275f3a589b54\" (UID: \"05ea8853-9e7c-487a-ac09-275f3a589b54\") "
Nov 02 17:55:29.128245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.128199    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj" pod "05ea8853-9e7c-487a-ac09-275f3a589b54" (UID: "05ea8853-9e7c-487a-ac09-275f3a589b54"). InnerVolumeSpecName "kube-api-access-gcgmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:29.128403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.128272    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:29.128974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.128950    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca" pod "05ea8853-9e7c-487a-ac09-275f3a589b54" (UID: "05ea8853-9e7c-487a-ac09-275f3a589b54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:29.230218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.230185    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05ea8853-9e7c-487a-ac09-275f3a589b54-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:29.230218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.230215    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcgmj\" (UniqueName: \"kubernetes.io/projected/05ea8853-9e7c-487a-ac09-275f3a589b54-kube-api-access-gcgmj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:29.922028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.921942    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"
Nov 02 17:55:29.922028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.921971    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw" event={"ID":"05ea8853-9e7c-487a-ac09-275f3a589b54","Type":"ContainerDied","Data":"e05d45f141e459614752f9df1f84aa14bd21b6a731fbce5f42ed8ba53f31910e"}
Nov 02 17:55:29.922214 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:29.922030    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05d45f141e459614752f9df1f84aa14bd21b6a731fbce5f42ed8ba53f31910e"
Nov 02 17:55:30.431348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.431314    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"]
Nov 02 17:55:30.431915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.431900    5042 cpu_manager.go:414] "containerMap: removing container" podUID="05ea8853-9e7c-487a-ac09-275f3a589b54" containerName="revoke-keys"
Nov 02 17:55:30.431990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.431916    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05ea8853-9e7c-487a-ac09-275f3a589b54" containerName="revoke-keys"
Nov 02 17:55:30.432062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.432052    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="05ea8853-9e7c-487a-ac09-275f3a589b54" containerName="revoke-keys"
Nov 02 17:55:30.435869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.435821    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.442772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.442742    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:55:30.442975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.442812    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:55:30.442975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.442922    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:55:30.444042 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.444016    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:55:30.448190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.448164    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"]
Nov 02 17:55:30.550359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.550321    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.550565 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.550495    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.550633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.550607    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47b8n\" (UniqueName: \"kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.652058 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.652013    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.652249 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.652120    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47b8n\" (UniqueName: \"kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.652415 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.652375    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.653072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.652733    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.655327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.655303    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.668714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.668681    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47b8n\" (UniqueName: \"kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n\") pod \"maas-api-revoke-sub-dce41582-w75f4\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:30.753157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:30.753118    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:31.068928 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:31.068893    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"]
Nov 02 17:55:31.069600 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:31.069572    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f37c283_4680_451d_ae2e_1cc801ca0768.slice/crio-18cc110a1f6997a3628730a54c332603960983b074d62d805b738a7424e75d97 WatchSource:0}: Error finding container 18cc110a1f6997a3628730a54c332603960983b074d62d805b738a7424e75d97: Status 404 returned error can't find the container with id 18cc110a1f6997a3628730a54c332603960983b074d62d805b738a7424e75d97
Nov 02 17:55:31.964055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:31.964016    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4" event={"ID":"9f37c283-4680-451d-ae2e-1cc801ca0768","Type":"ContainerStarted","Data":"b19cf4a6c3798edfc075a046cc51a45de674717119290db9f4bcb3ddb4dcb98f"}
Nov 02 17:55:31.964470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:31.964074    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4" event={"ID":"9f37c283-4680-451d-ae2e-1cc801ca0768","Type":"ContainerStarted","Data":"18cc110a1f6997a3628730a54c332603960983b074d62d805b738a7424e75d97"}
Nov 02 17:55:31.992360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:31.992292    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4" podStartSLOduration=1.992274499 podStartE2EDuration="1.992274499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:31.992208055 +0000 UTC m=+1344.571617442" watchObservedRunningTime="2026-11-02 17:55:31.992274499 +0000 UTC m=+1344.571683889"
Nov 02 17:55:32.615868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.614334    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"]
Nov 02 17:55:32.633066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.633024    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"]
Nov 02 17:55:32.633257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.633159    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.647680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.645211    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-47e1c2-openshift-default-dockercfg-q8nvl\""
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720276    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720356    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720436    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720485    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7lp6\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720528    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720588    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720629    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720672    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.728867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.720725    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.821866    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.821950    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822038    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822091    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7lp6\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822137    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822200    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822245    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822286    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.822659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.822345    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.824564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.824513    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.825065 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.824901    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.825468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.825423    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.826115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.826093    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.826506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.826488    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.897822 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.897709    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7lp6\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.899394 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.898433    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.902858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.902261    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.908863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.908463    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token\") pod \"e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") " pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:32.962225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:32.962189    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:33.008358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.008181    5042 generic.go:356] "Generic (PLEG): container finished" podID="9f37c283-4680-451d-ae2e-1cc801ca0768" containerID="b19cf4a6c3798edfc075a046cc51a45de674717119290db9f4bcb3ddb4dcb98f" exitCode=0
Nov 02 17:55:33.008358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.008315    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4" event={"ID":"9f37c283-4680-451d-ae2e-1cc801ca0768","Type":"ContainerDied","Data":"b19cf4a6c3798edfc075a046cc51a45de674717119290db9f4bcb3ddb4dcb98f"}
Nov 02 17:55:33.345478 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:33.345447    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fefb15_6fe5_42c9_b808_cc4fc34eae70.slice/crio-4c4d80b39f32f4186791b57489611767de2e8044ca52fd0aa5f78f0f5e437402 WatchSource:0}: Error finding container 4c4d80b39f32f4186791b57489611767de2e8044ca52fd0aa5f78f0f5e437402: Status 404 returned error can't find the container with id 4c4d80b39f32f4186791b57489611767de2e8044ca52fd0aa5f78f0f5e437402
Nov 02 17:55:33.345632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.345532    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"]
Nov 02 17:55:33.348145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.348117    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:33.348236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.348179    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:33.348236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.348209    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:33.497827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.497787    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"]
Nov 02 17:55:33.503364 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.503337    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.523222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.523190    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"]
Nov 02 17:55:33.538895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.538724    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.539139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.538895    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.539139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.538937    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw568\" (UniqueName: \"kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.640783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.640695    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.640783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.640735    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw568\" (UniqueName: \"kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.641068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.640936    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.641560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.641532    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.643497 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.643476    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.653379 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.653348    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw568\" (UniqueName: \"kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568\") pod \"maas-api-revoke-sub-0ae5731b-6nqx5\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:33.829062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:33.829022    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:34.050757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.050715    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" event={"ID":"95fefb15-6fe5-42c9-b808-cc4fc34eae70","Type":"ContainerStarted","Data":"d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"}
Nov 02 17:55:34.051284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.050785    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" event={"ID":"95fefb15-6fe5-42c9-b808-cc4fc34eae70","Type":"ContainerStarted","Data":"4c4d80b39f32f4186791b57489611767de2e8044ca52fd0aa5f78f0f5e437402"}
Nov 02 17:55:34.112765 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.110004    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podStartSLOduration=2.109983197 podStartE2EDuration="2.109983197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:34.090049807 +0000 UTC m=+1346.669459194" watchObservedRunningTime="2026-11-02 17:55:34.109983197 +0000 UTC m=+1346.689392583"
Nov 02 17:55:34.129220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.129181    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"]
Nov 02 17:55:34.139885 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:34.136943    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2389da41_ad7e_4674_90ce_79214e405ac0.slice/crio-9fd7656769b4f703596e4c7c9fdcb7e125d8f8a773eecb503cd4c14b87c655cd WatchSource:0}: Error finding container 9fd7656769b4f703596e4c7c9fdcb7e125d8f8a773eecb503cd4c14b87c655cd: Status 404 returned error can't find the container with id 9fd7656769b4f703596e4c7c9fdcb7e125d8f8a773eecb503cd4c14b87c655cd
Nov 02 17:55:34.282164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.278985    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:34.350922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.350870    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca\") pod \"9f37c283-4680-451d-ae2e-1cc801ca0768\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") "
Nov 02 17:55:34.351107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.351069    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca\") pod \"9f37c283-4680-451d-ae2e-1cc801ca0768\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") "
Nov 02 17:55:34.351178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.351153    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n\" (UniqueName: \"kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n\") pod \"9f37c283-4680-451d-ae2e-1cc801ca0768\" (UID: \"9f37c283-4680-451d-ae2e-1cc801ca0768\") "
Nov 02 17:55:34.351313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.351281    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca" pod "9f37c283-4680-451d-ae2e-1cc801ca0768" (UID: "9f37c283-4680-451d-ae2e-1cc801ca0768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:34.351795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.351769    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:34.353518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.353490    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca" pod "9f37c283-4680-451d-ae2e-1cc801ca0768" (UID: "9f37c283-4680-451d-ae2e-1cc801ca0768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:34.353518 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.353495    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n" pod "9f37c283-4680-451d-ae2e-1cc801ca0768" (UID: "9f37c283-4680-451d-ae2e-1cc801ca0768"). InnerVolumeSpecName "kube-api-access-47b8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:34.453511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.453476    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f37c283-4680-451d-ae2e-1cc801ca0768-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:34.453511 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.453508    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47b8n\" (UniqueName: \"kubernetes.io/projected/9f37c283-4680-451d-ae2e-1cc801ca0768-kube-api-access-47b8n\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:34.971175 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.971139    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:34.979875 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:34.979147    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:35.176028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.173793    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4" event={"ID":"9f37c283-4680-451d-ae2e-1cc801ca0768","Type":"ContainerDied","Data":"18cc110a1f6997a3628730a54c332603960983b074d62d805b738a7424e75d97"}
Nov 02 17:55:35.176028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.173883    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18cc110a1f6997a3628730a54c332603960983b074d62d805b738a7424e75d97"
Nov 02 17:55:35.176028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.173967    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"
Nov 02 17:55:35.188532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.188176    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5" event={"ID":"2389da41-ad7e-4674-90ce-79214e405ac0","Type":"ContainerStarted","Data":"f500c566e93b9ad02620af945ca5024495fd6e3dc4b1049a66a49631dab0b392"}
Nov 02 17:55:35.188532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.188229    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5" event={"ID":"2389da41-ad7e-4674-90ce-79214e405ac0","Type":"ContainerStarted","Data":"9fd7656769b4f703596e4c7c9fdcb7e125d8f8a773eecb503cd4c14b87c655cd"}
Nov 02 17:55:35.188532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.188259    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:35.192590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.192571    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:55:35.213936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:35.213878    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5" podStartSLOduration=2.213857481 podStartE2EDuration="2.213857481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:35.209176079 +0000 UTC m=+1347.788585464" watchObservedRunningTime="2026-11-02 17:55:35.213857481 +0000 UTC m=+1347.793266864"
Nov 02 17:55:36.211185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:36.211143    5042 generic.go:356] "Generic (PLEG): container finished" podID="2389da41-ad7e-4674-90ce-79214e405ac0" containerID="f500c566e93b9ad02620af945ca5024495fd6e3dc4b1049a66a49631dab0b392" exitCode=0
Nov 02 17:55:36.211795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:36.211226    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5" event={"ID":"2389da41-ad7e-4674-90ce-79214e405ac0","Type":"ContainerDied","Data":"f500c566e93b9ad02620af945ca5024495fd6e3dc4b1049a66a49631dab0b392"}
Nov 02 17:55:37.410103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.410078    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:37.547494 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.547410    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca\") pod \"2389da41-ad7e-4674-90ce-79214e405ac0\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") "
Nov 02 17:55:37.547659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.547532    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca\") pod \"2389da41-ad7e-4674-90ce-79214e405ac0\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") "
Nov 02 17:55:37.547659 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.547591    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568\" (UniqueName: \"kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568\") pod \"2389da41-ad7e-4674-90ce-79214e405ac0\" (UID: \"2389da41-ad7e-4674-90ce-79214e405ac0\") "
Nov 02 17:55:37.547958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.547932    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca" pod "2389da41-ad7e-4674-90ce-79214e405ac0" (UID: "2389da41-ad7e-4674-90ce-79214e405ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:37.548248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.548211    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:37.549676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.549655    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca" pod "2389da41-ad7e-4674-90ce-79214e405ac0" (UID: "2389da41-ad7e-4674-90ce-79214e405ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:37.549793 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.549774    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568" pod "2389da41-ad7e-4674-90ce-79214e405ac0" (UID: "2389da41-ad7e-4674-90ce-79214e405ac0"). InnerVolumeSpecName "kube-api-access-cw568". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:37.650176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.650140    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2389da41-ad7e-4674-90ce-79214e405ac0-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:37.650176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:37.650170    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw568\" (UniqueName: \"kubernetes.io/projected/2389da41-ad7e-4674-90ce-79214e405ac0-kube-api-access-cw568\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:38.239790 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.239713    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"]
Nov 02 17:55:38.241023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.240995    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9f37c283-4680-451d-ae2e-1cc801ca0768" containerName="revoke-keys"
Nov 02 17:55:38.241230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.241215    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f37c283-4680-451d-ae2e-1cc801ca0768" containerName="revoke-keys"
Nov 02 17:55:38.241480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.241463    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2389da41-ad7e-4674-90ce-79214e405ac0" containerName="revoke-keys"
Nov 02 17:55:38.241653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.241639    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2389da41-ad7e-4674-90ce-79214e405ac0" containerName="revoke-keys"
Nov 02 17:55:38.242171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.242049    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f37c283-4680-451d-ae2e-1cc801ca0768" containerName="revoke-keys"
Nov 02 17:55:38.242314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.242301    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2389da41-ad7e-4674-90ce-79214e405ac0" containerName="revoke-keys"
Nov 02 17:55:38.329186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.329154    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.341866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.334398    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5" event={"ID":"2389da41-ad7e-4674-90ce-79214e405ac0","Type":"ContainerDied","Data":"9fd7656769b4f703596e4c7c9fdcb7e125d8f8a773eecb503cd4c14b87c655cd"}
Nov 02 17:55:38.341866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.334457    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fd7656769b4f703596e4c7c9fdcb7e125d8f8a773eecb503cd4c14b87c655cd"
Nov 02 17:55:38.341866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.334506    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"
Nov 02 17:55:38.360954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.355924    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"]
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.377561    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.377636    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8msvh\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.377683    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.377803    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.377905    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.377985    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.378045    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.378077    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.388050 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.378100    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.482153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.482119    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.540614 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.540501    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.541348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.541322    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.541999 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.541980    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.545651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.545629    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.546197 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.546177    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.547064 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.547046    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.549081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.541879    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.549223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.546940    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.549345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.509831    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.549345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.542386    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.550193 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.550170    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8msvh\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.555537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.555509    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.556051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.556028    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.579081 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.579019    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.580077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.579998    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.582985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.582922    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8msvh\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.586373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.586326    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy\") pod \"e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") " pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:38.720427 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:38.720389    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:39.058623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.058479    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"]
Nov 02 17:55:39.061152 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:39.061098    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb7660e_88b1_4348_9076_2a863a400210.slice/crio-6e6a66892c04ce63a40785115cd3726e3c87a71b98b293a18bc892cbb6b49cfd WatchSource:0}: Error finding container 6e6a66892c04ce63a40785115cd3726e3c87a71b98b293a18bc892cbb6b49cfd: Status 404 returned error can't find the container with id 6e6a66892c04ce63a40785115cd3726e3c87a71b98b293a18bc892cbb6b49cfd
Nov 02 17:55:39.063820 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.063783    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:39.063988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.063883    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:39.063988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.063931    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:39.350975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.350779    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" event={"ID":"7fb7660e-88b1-4348-9076-2a863a400210","Type":"ContainerStarted","Data":"7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b"}
Nov 02 17:55:39.350975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.350880    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" event={"ID":"7fb7660e-88b1-4348-9076-2a863a400210","Type":"ContainerStarted","Data":"6e6a66892c04ce63a40785115cd3726e3c87a71b98b293a18bc892cbb6b49cfd"}
Nov 02 17:55:39.394817 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.394758    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podStartSLOduration=1.394738264 podStartE2EDuration="1.394738264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:39.376305793 +0000 UTC m=+1351.955715190" watchObservedRunningTime="2026-11-02 17:55:39.394738264 +0000 UTC m=+1351.974147652"
Nov 02 17:55:39.725881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:39.724970    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:40.719861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.719660    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"]
Nov 02 17:55:40.722735 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.722707    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 17:55:40.722916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.722756    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 17:55:40.724280 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.724259    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.728657 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.728633    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:55:40.734831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.734275    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:55:40.734831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.734766    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:55:40.734831 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.734791    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:55:40.740807 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.740783    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"]
Nov 02 17:55:40.819863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.819808    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdb6\" (UniqueName: \"kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.820121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.819965    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.820121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.820056    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.921813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.921780    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdb6\" (UniqueName: \"kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.922033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.921879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.922033 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.921944    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.922787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.922757    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.924521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.924498    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:40.931781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:40.931758    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdb6\" (UniqueName: \"kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6\") pod \"maas-api-revoke-sub-8c186922-b82pb\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:41.042261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:41.042177    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:41.371367 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:41.371334    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"]
Nov 02 17:55:41.371522 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:41.371498    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9f21ab_3e61_407a_b465_213b7658fd9e.slice/crio-57cb7b95d747d5f69a43a5ef4d9f6aac7acd847cc77c99e432ee9b8c58080052 WatchSource:0}: Error finding container 57cb7b95d747d5f69a43a5ef4d9f6aac7acd847cc77c99e432ee9b8c58080052: Status 404 returned error can't find the container with id 57cb7b95d747d5f69a43a5ef4d9f6aac7acd847cc77c99e432ee9b8c58080052
Nov 02 17:55:41.721480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:41.721442    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 17:55:41.721677 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:41.721538    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 17:55:42.390865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:42.386667    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb" event={"ID":"cc9f21ab-3e61-407a-b465-213b7658fd9e","Type":"ContainerStarted","Data":"7f9a5a451c8868fc8f9a41ecec252742f92b637c52160e11c65c71293ddd75d5"}
Nov 02 17:55:42.390865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:42.386738    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb" event={"ID":"cc9f21ab-3e61-407a-b465-213b7658fd9e","Type":"ContainerStarted","Data":"57cb7b95d747d5f69a43a5ef4d9f6aac7acd847cc77c99e432ee9b8c58080052"}
Nov 02 17:55:42.421465 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:42.421406    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb" podStartSLOduration=2.4213912730000002 podStartE2EDuration="2.421391273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:42.418491016 +0000 UTC m=+1354.997900403" watchObservedRunningTime="2026-11-02 17:55:42.421391273 +0000 UTC m=+1355.000800636"
Nov 02 17:55:42.721434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:42.721395    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 17:55:42.721618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:42.721485    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 17:55:43.417892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:43.411892    5042 generic.go:356] "Generic (PLEG): container finished" podID="cc9f21ab-3e61-407a-b465-213b7658fd9e" containerID="7f9a5a451c8868fc8f9a41ecec252742f92b637c52160e11c65c71293ddd75d5" exitCode=0
Nov 02 17:55:43.417892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:43.412012    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb" event={"ID":"cc9f21ab-3e61-407a-b465-213b7658fd9e","Type":"ContainerDied","Data":"7f9a5a451c8868fc8f9a41ecec252742f92b637c52160e11c65c71293ddd75d5"}
Nov 02 17:55:43.721696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:43.721657    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 17:55:43.721930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:43.721719    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 17:55:44.732904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.732167    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 17:55:44.732904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.732229    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 17:55:44.831128 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.830750    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:44.955115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.955081    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6\" (UniqueName: \"kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6\") pod \"cc9f21ab-3e61-407a-b465-213b7658fd9e\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") "
Nov 02 17:55:44.955471 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.955447    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca\") pod \"cc9f21ab-3e61-407a-b465-213b7658fd9e\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") "
Nov 02 17:55:44.955830 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.955811    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca\") pod \"cc9f21ab-3e61-407a-b465-213b7658fd9e\" (UID: \"cc9f21ab-3e61-407a-b465-213b7658fd9e\") "
Nov 02 17:55:44.956877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.955817    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca" pod "cc9f21ab-3e61-407a-b465-213b7658fd9e" (UID: "cc9f21ab-3e61-407a-b465-213b7658fd9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:44.958824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.958796    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca" pod "cc9f21ab-3e61-407a-b465-213b7658fd9e" (UID: "cc9f21ab-3e61-407a-b465-213b7658fd9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:44.958824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:44.958807    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6" pod "cc9f21ab-3e61-407a-b465-213b7658fd9e" (UID: "cc9f21ab-3e61-407a-b465-213b7658fd9e"). InnerVolumeSpecName "kube-api-access-jkdb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:45.058124 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.058039    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:45.058124 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.058071    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc9f21ab-3e61-407a-b465-213b7658fd9e-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:45.058124 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.058093    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdb6\" (UniqueName: \"kubernetes.io/projected/cc9f21ab-3e61-407a-b465-213b7658fd9e-kube-api-access-jkdb6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:55:45.432381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.432300    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb" event={"ID":"cc9f21ab-3e61-407a-b465-213b7658fd9e","Type":"ContainerDied","Data":"57cb7b95d747d5f69a43a5ef4d9f6aac7acd847cc77c99e432ee9b8c58080052"}
Nov 02 17:55:45.432381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.432320    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"
Nov 02 17:55:45.432381 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.432353    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57cb7b95d747d5f69a43a5ef4d9f6aac7acd847cc77c99e432ee9b8c58080052"
Nov 02 17:55:45.594951 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.594917    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"]
Nov 02 17:55:45.596166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.596140    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cc9f21ab-3e61-407a-b465-213b7658fd9e" containerName="revoke-keys"
Nov 02 17:55:45.596323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.596311    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc9f21ab-3e61-407a-b465-213b7658fd9e" containerName="revoke-keys"
Nov 02 17:55:45.596542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.596531    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc9f21ab-3e61-407a-b465-213b7658fd9e" containerName="revoke-keys"
Nov 02 17:55:45.600935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.600915    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.610506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.610459    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-86f5f0-openshift-default-dockercfg-lwhdl\""
Nov 02 17:55:45.643821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.643775    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"]
Nov 02 17:55:45.721054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.721015    5042 patch_prober.go:29] interesting pod/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 17:55:45.721222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.721084    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.781802    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.781881    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.781977    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.782010    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.782061    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.782113    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.782180    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjcj\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.782247    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.793866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.782302    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890042    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890182    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890215    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890304    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890337    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890386    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890437    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890507    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjcj\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.896974 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.890573    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.912866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.900712    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.912866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.901014    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.912866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.901284    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.912866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.901533    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.912866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.901583    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.912866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.907317    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.936017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.935357    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.951237 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.950140    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:45.966879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:45.955628    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjcj\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj\") pod \"e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") " pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:46.254428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:46.254396    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:46.596185 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:46.596128    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf9580b_aed3_4788_a8f9_fb1427610c9a.slice/crio-91760b23738c91107a2cca3a6a11dd2dc2e5bc67961ad9dc9a3c9f3f32b53950 WatchSource:0}: Error finding container 91760b23738c91107a2cca3a6a11dd2dc2e5bc67961ad9dc9a3c9f3f32b53950: Status 404 returned error can't find the container with id 91760b23738c91107a2cca3a6a11dd2dc2e5bc67961ad9dc9a3c9f3f32b53950
Nov 02 17:55:46.599428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:46.599393    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:46.599564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:46.599479    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:46.599564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:46.599525    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:46.600024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:46.599990    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"]
Nov 02 17:55:47.166743 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:47.166706    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:47.168231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:47.167309    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:47.168458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:47.168405    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 17:55:47.555999 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:47.555897    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" event={"ID":"9cf9580b-aed3-4788-a8f9-fb1427610c9a","Type":"ContainerStarted","Data":"546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"}
Nov 02 17:55:47.555999 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:47.555964    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" event={"ID":"9cf9580b-aed3-4788-a8f9-fb1427610c9a","Type":"ContainerStarted","Data":"91760b23738c91107a2cca3a6a11dd2dc2e5bc67961ad9dc9a3c9f3f32b53950"}
Nov 02 17:55:47.586593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:47.586539    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podStartSLOduration=2.586523875 podStartE2EDuration="2.586523875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:47.586492286 +0000 UTC m=+1360.165901674" watchObservedRunningTime="2026-11-02 17:55:47.586523875 +0000 UTC m=+1360.165933260"
Nov 02 17:55:48.255008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:48.254976    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:48.261158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:48.261133    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:48.541420 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:55:48.541322    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:55:48.562495 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:48.562465    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:48.563360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:48.563338    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:55:53.561136 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.561100    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"]
Nov 02 17:55:53.567456 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.567430    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.577348 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.577321    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b6b6f4-openshift-default-dockercfg-rnffh\""
Nov 02 17:55:53.587806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.587774    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"]
Nov 02 17:55:53.753204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.752923    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.774097 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.774065    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.779961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.779925    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.780262 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.780240    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.780433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.780416    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbv7m\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.793863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.780592    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.794289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.794257    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.794499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.794483    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.794692 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.794675    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.896965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.896869    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.904410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.904380    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.905464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.905439    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.905358    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.916274    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.897559    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.917371    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917619 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.917589    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.917646    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.917671    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.917736    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.917865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.917793    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbv7m\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.918983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.918560    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.919359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.918571    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.954437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.954368    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.960150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.960061    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbv7m\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.975817 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.975283    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:53.979372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:53.979339    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:54.197564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.197523    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:54.522096 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:54.522063    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3aadd4_5c54_4a81_ab70_8258d5e042ee.slice/crio-4eeb48a7449363467c3ece9e904d98d242ab1f6b6a37d46478d91d6b75b7dd35 WatchSource:0}: Error finding container 4eeb48a7449363467c3ece9e904d98d242ab1f6b6a37d46478d91d6b75b7dd35: Status 404 returned error can't find the container with id 4eeb48a7449363467c3ece9e904d98d242ab1f6b6a37d46478d91d6b75b7dd35
Nov 02 17:55:54.523000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.522921    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"]
Nov 02 17:55:54.525166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.525133    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:54.525261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.525199    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:54.525261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.525231    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:54.651772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.651733    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" event={"ID":"4a3aadd4-5c54-4a81-ab70-8258d5e042ee","Type":"ContainerStarted","Data":"d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"}
Nov 02 17:55:54.652227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.651800    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" event={"ID":"4a3aadd4-5c54-4a81-ab70-8258d5e042ee","Type":"ContainerStarted","Data":"4eeb48a7449363467c3ece9e904d98d242ab1f6b6a37d46478d91d6b75b7dd35"}
Nov 02 17:55:54.682866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:54.682690    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podStartSLOduration=1.6826708799999999 podStartE2EDuration="1.68267088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:54.679464789 +0000 UTC m=+1367.258874204" watchObservedRunningTime="2026-11-02 17:55:54.68267088 +0000 UTC m=+1367.262080267"
Nov 02 17:55:55.198268 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:55.198189    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:55.224564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:55.224231    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:55.659601 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:55.659516    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:55.660539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:55.660517    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:55:57.026900 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.026864    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"]
Nov 02 17:55:57.031418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.031401    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.040377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.040350    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:55:57.040521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.040406    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:55:57.040521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.040498    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:55:57.040631 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.040589    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:55:57.045805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.045779    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"]
Nov 02 17:55:57.094707 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.094670    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.094973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.094815    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kq4n\" (UniqueName: \"kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.095055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.095022    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.196306 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.196272    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kq4n\" (UniqueName: \"kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.196523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.196484    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.196594 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.196565    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.197374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.197344    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.199315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.199291    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.232000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.231965    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kq4n\" (UniqueName: \"kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n\") pod \"maas-api-revoke-sub-0545acec-6hxkb\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.348226 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.348132    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:55:57.656743 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:55:57.656706    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2c1132_690f_4153_9184_1feae3dea181.slice/crio-7aa1cee01eb1a3a0d1b36f5cc8b81a17d5f5fa516e063738c6748a36b564a137 WatchSource:0}: Error finding container 7aa1cee01eb1a3a0d1b36f5cc8b81a17d5f5fa516e063738c6748a36b564a137: Status 404 returned error can't find the container with id 7aa1cee01eb1a3a0d1b36f5cc8b81a17d5f5fa516e063738c6748a36b564a137
Nov 02 17:55:57.657949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.657908    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"]
Nov 02 17:55:57.730211 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:57.730177    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb" event={"ID":"fc2c1132-690f-4153-9184-1feae3dea181","Type":"ContainerStarted","Data":"7aa1cee01eb1a3a0d1b36f5cc8b81a17d5f5fa516e063738c6748a36b564a137"}
Nov 02 17:55:58.850827 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:58.848678    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb" event={"ID":"fc2c1132-690f-4153-9184-1feae3dea181","Type":"ContainerStarted","Data":"87ef3ecbe2f482f9404dc05c2f0a74427ca846e6b5506ef43b9a40c04cdc6cc1"}
Nov 02 17:55:58.909642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:58.905192    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb" podStartSLOduration=2.9051687250000002 podStartE2EDuration="2.905168725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:58.899712472 +0000 UTC m=+1371.479121860" watchObservedRunningTime="2026-11-02 17:55:58.905168725 +0000 UTC m=+1371.484578112"
Nov 02 17:55:59.913732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:59.913038    5042 generic.go:356] "Generic (PLEG): container finished" podID="fc2c1132-690f-4153-9184-1feae3dea181" containerID="87ef3ecbe2f482f9404dc05c2f0a74427ca846e6b5506ef43b9a40c04cdc6cc1" exitCode=0
Nov 02 17:55:59.913732 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:55:59.913302    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb" event={"ID":"fc2c1132-690f-4153-9184-1feae3dea181","Type":"ContainerDied","Data":"87ef3ecbe2f482f9404dc05c2f0a74427ca846e6b5506ef43b9a40c04cdc6cc1"}
Nov 02 17:56:01.037022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.036985    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"]
Nov 02 17:56:01.061651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.061484    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"]
Nov 02 17:56:01.156307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.082214    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.156759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.156069    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.163628 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.163495    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.164104 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.164076    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.423158 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.180574    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.423475 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.306574    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1e2281-openshift-default-dockercfg-xhl4q\""
Nov 02 17:56:01.423761 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.398647    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:01.424625 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.423653    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.425121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.425043    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.425298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.425281    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.425767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.425620    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.440727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.426034    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srh6n\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.603453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.603413    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.603867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.603824    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.606992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.606966    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.607195 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.607177    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.607329 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.607314    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.607464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.607449    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.607567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.607553    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.607739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.607703    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.607904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.607877    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srh6n\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.612367 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.612342    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.612623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.612597    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.612894 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.612870    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.613530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.613506    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.614098 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.614072    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.646936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.637234    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.646936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.645064    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.646936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.646429    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded"
Nov 02 17:56:01.651670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.650925    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srh6n\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.657998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.657958    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:01.716420 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.716358    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:01.726155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:01.726122    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:02.096865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.082077    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:56:02.166362 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.166329    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca\") pod \"fc2c1132-690f-4153-9184-1feae3dea181\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") "
Nov 02 17:56:02.166555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.166446    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n\" (UniqueName: \"kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n\") pod \"fc2c1132-690f-4153-9184-1feae3dea181\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") "
Nov 02 17:56:02.166555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.166479    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca\") pod \"fc2c1132-690f-4153-9184-1feae3dea181\" (UID: \"fc2c1132-690f-4153-9184-1feae3dea181\") "
Nov 02 17:56:02.167178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.167141    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca" pod "fc2c1132-690f-4153-9184-1feae3dea181" (UID: "fc2c1132-690f-4153-9184-1feae3dea181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:02.169640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.169612    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca" pod "fc2c1132-690f-4153-9184-1feae3dea181" (UID: "fc2c1132-690f-4153-9184-1feae3dea181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:56:02.171463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.171436    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n" pod "fc2c1132-690f-4153-9184-1feae3dea181" (UID: "fc2c1132-690f-4153-9184-1feae3dea181"). InnerVolumeSpecName "kube-api-access-9kq4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:02.303865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.300465    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:02.303865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.300499    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kq4n\" (UniqueName: \"kubernetes.io/projected/fc2c1132-690f-4153-9184-1feae3dea181-kube-api-access-9kq4n\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:02.303865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.300514    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2c1132-690f-4153-9184-1feae3dea181-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:02.410594 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:56:02.410547    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f975ec_36ec_401d_b5a8_cca7fcc299fb.slice/crio-953d06aac9a7902ebfbd11d87014ccbbf4df47ac70cc6de8f9f56de188cf7837 WatchSource:0}: Error finding container 953d06aac9a7902ebfbd11d87014ccbbf4df47ac70cc6de8f9f56de188cf7837: Status 404 returned error can't find the container with id 953d06aac9a7902ebfbd11d87014ccbbf4df47ac70cc6de8f9f56de188cf7837
Nov 02 17:56:02.418142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.418062    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:02.418490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.418462    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:02.418785 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.418765    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:02.433753 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:02.433702    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"]
Nov 02 17:56:03.053324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.053290    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"
Nov 02 17:56:03.053550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.053322    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb" event={"ID":"fc2c1132-690f-4153-9184-1feae3dea181","Type":"ContainerDied","Data":"7aa1cee01eb1a3a0d1b36f5cc8b81a17d5f5fa516e063738c6748a36b564a137"}
Nov 02 17:56:03.053550 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.053369    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aa1cee01eb1a3a0d1b36f5cc8b81a17d5f5fa516e063738c6748a36b564a137"
Nov 02 17:56:03.055644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.055604    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" event={"ID":"a7f975ec-36ec-401d-b5a8-cca7fcc299fb","Type":"ContainerStarted","Data":"3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"}
Nov 02 17:56:03.055791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.055666    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" event={"ID":"a7f975ec-36ec-401d-b5a8-cca7fcc299fb","Type":"ContainerStarted","Data":"953d06aac9a7902ebfbd11d87014ccbbf4df47ac70cc6de8f9f56de188cf7837"}
Nov 02 17:56:03.131407 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.131208    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podStartSLOduration=3.131189746 podStartE2EDuration="3.131189746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:03.125414259 +0000 UTC m=+1375.704823645" watchObservedRunningTime="2026-11-02 17:56:03.131189746 +0000 UTC m=+1375.710599133"
Nov 02 17:56:03.727499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.727466    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:03.733564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:03.733529    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:04.065774 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:04.065684    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:04.067334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:04.067291    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:56:07.851870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.851018    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"]
Nov 02 17:56:07.852455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.852014    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fc2c1132-690f-4153-9184-1feae3dea181" containerName="revoke-keys"
Nov 02 17:56:07.852455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.852032    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2c1132-690f-4153-9184-1feae3dea181" containerName="revoke-keys"
Nov 02 17:56:07.852455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.852170    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2c1132-690f-4153-9184-1feae3dea181" containerName="revoke-keys"
Nov 02 17:56:07.858210 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.858182    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.879241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.879209    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"]
Nov 02 17:56:07.898598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.898562    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9165c3-openshift-default-dockercfg-7wchl\""
Nov 02 17:56:07.980998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.980949    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rzv\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981083    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981157    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981274    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981303    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981355 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981329    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981597 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981442    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981593    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:07.981661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:07.981637    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087379    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087456    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087528    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rzv\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087584    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087637    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087713    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087757    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087785    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.087873    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.088882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.088433    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.089778 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.089706    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.097576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.097160    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.097576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.097506    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.098110 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.098064    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.099750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.099720    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.118914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.117557    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.119805 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.119740    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.133863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.130941    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rzv\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.189901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.186940    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:08.730118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.730087    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"]
Nov 02 17:56:08.732762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.732718    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:08.732967 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.732807    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:08.733071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:08.733047    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:09.205909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:09.204607    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" event={"ID":"db2d4865-b81a-4d68-9dde-1bc29f58bde7","Type":"ContainerStarted","Data":"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"}
Nov 02 17:56:09.205909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:09.204710    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" event={"ID":"db2d4865-b81a-4d68-9dde-1bc29f58bde7","Type":"ContainerStarted","Data":"f58ba23bd43d181eb3924b8ce95b816995fe3662b44e9bfea954a9712a0a6739"}
Nov 02 17:56:09.242652 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:09.242580    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podStartSLOduration=2.242563945 podStartE2EDuration="2.242563945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:09.23945062 +0000 UTC m=+1381.818860007" watchObservedRunningTime="2026-11-02 17:56:09.242563945 +0000 UTC m=+1381.821973331"
Nov 02 17:56:10.188332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:10.188296    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:10.195174 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:10.195143    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:10.214985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:10.214940    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:10.216150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:10.216127    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:56:11.227821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:11.227785    5042 generic.go:356] "Generic (PLEG): container finished" podID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerID="d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8" exitCode=0
Nov 02 17:56:11.228399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:11.227895    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" event={"ID":"95fefb15-6fe5-42c9-b808-cc4fc34eae70","Type":"ContainerDied","Data":"d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"}
Nov 02 17:56:11.228399 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:11.228355    5042 scope.go:122] "RemoveContainer" containerID="d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"
Nov 02 17:56:11.230056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:11.230028    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:11.230179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:11.230096    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:11.230179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:11.230128    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:12.238697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:12.238661    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" event={"ID":"95fefb15-6fe5-42c9-b808-cc4fc34eae70","Type":"ContainerStarted","Data":"3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5"}
Nov 02 17:56:12.963046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:12.963010    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:56:12.963344 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:12.963323    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 17:56:12.963498 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:12.963477    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 17:56:13.349089 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.348991    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"]
Nov 02 17:56:13.631492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.631370    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.647868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.646119    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"]
Nov 02 17:56:13.653182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.653152    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-9165c3\""
Nov 02 17:56:13.658338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.658316    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-9165c3-dockercfg-vl5w8\""
Nov 02 17:56:13.753779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.753738    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pwn6\" (UniqueName: \"kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6\") pod \"payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.753985 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.753859    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.855749 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.855715    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwn6\" (UniqueName: \"kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6\") pod \"payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.855960 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.855809    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.856558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.856537    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.865312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.865280    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwn6\" (UniqueName: \"kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6\") pod \"payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.966863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.959350    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:13.970453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.970014    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 17:56:13.970453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:13.970061    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 17:56:14.323776 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:14.323741    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"]
Nov 02 17:56:14.325514 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:56:14.325475    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c0ac11_411b_487c_afd9_abe76d467bba.slice/crio-1680e605a51b398da50b78f0a7241644cbca7f132e6df016ba80aa554639ad30 WatchSource:0}: Error finding container 1680e605a51b398da50b78f0a7241644cbca7f132e6df016ba80aa554639ad30: Status 404 returned error can't find the container with id 1680e605a51b398da50b78f0a7241644cbca7f132e6df016ba80aa554639ad30
Nov 02 17:56:14.419313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:14.419277    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" event={"ID":"a8c0ac11-411b-487c-afd9-abe76d467bba","Type":"ContainerStarted","Data":"1680e605a51b398da50b78f0a7241644cbca7f132e6df016ba80aa554639ad30"}
Nov 02 17:56:14.963390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:14.963357    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 17:56:14.963598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:14.963410    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 17:56:15.429566 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:15.429469    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" event={"ID":"a8c0ac11-411b-487c-afd9-abe76d467bba","Type":"ContainerStarted","Data":"27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd"}
Nov 02 17:56:15.430051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:15.429563    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:15.454361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:15.454301    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" podStartSLOduration=2.454282928 podStartE2EDuration="2.454282928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:15.450259864 +0000 UTC m=+1388.029669252" watchObservedRunningTime="2026-11-02 17:56:15.454282928 +0000 UTC m=+1388.033692305"
Nov 02 17:56:15.963080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:15.963042    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 17:56:15.963284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:15.963097    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 17:56:16.128524 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.128455    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"]
Nov 02 17:56:16.141054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.136169    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.143016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.141669    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"]
Nov 02 17:56:16.144322 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.144276    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b1a788-openshift-default-dockercfg-kv6gv\""
Nov 02 17:56:16.312952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.272928    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.389070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.389031    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.389349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.389324    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.389581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.389557    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.389812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.389797    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.389979 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.389961    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.390178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.390146    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.390432 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.390409    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnwt\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.390606 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.390591    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.581726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.581622    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.582335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.582307    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.582484 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.582466    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.582711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.582691    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.582980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.582959    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.583137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.583121    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.583290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.583276    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.583462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.583445    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnwt\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.583603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.583591    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.584458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.584436    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.584825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.584807    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.585225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.585207    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.586346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.586323    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.599216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.599164    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.697815 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.677728    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnwt\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.705868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.697819    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.759080 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.759041    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.759371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.759348    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.759696 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.759671    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:16.964901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.963486    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 17:56:16.964901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:16.963570    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 17:56:17.116215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.116169    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"]
Nov 02 17:56:17.117293 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:56:17.117265    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a5e54c_3842_4947_859a_a4d1780467c6.slice/crio-c56f959152537f76cfddc471aaabcc5b51ff5fadb26740d001d870496a25e738 WatchSource:0}: Error finding container c56f959152537f76cfddc471aaabcc5b51ff5fadb26740d001d870496a25e738: Status 404 returned error can't find the container with id c56f959152537f76cfddc471aaabcc5b51ff5fadb26740d001d870496a25e738
Nov 02 17:56:17.120462 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.120426    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:17.120562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.120505    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:17.120562 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.120535    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:17.760623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.760405    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" event={"ID":"61a5e54c-3842-4947-859a-a4d1780467c6","Type":"ContainerStarted","Data":"999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"}
Nov 02 17:56:17.760623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.760464    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" event={"ID":"61a5e54c-3842-4947-859a-a4d1780467c6","Type":"ContainerStarted","Data":"c56f959152537f76cfddc471aaabcc5b51ff5fadb26740d001d870496a25e738"}
Nov 02 17:56:17.797862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.797739    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podStartSLOduration=1.797721407 podStartE2EDuration="1.797721407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:17.789403326 +0000 UTC m=+1390.368812713" watchObservedRunningTime="2026-11-02 17:56:17.797721407 +0000 UTC m=+1390.377130793"
Nov 02 17:56:17.972870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.970822    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body=
Nov 02 17:56:17.972870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:17.970918    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused"
Nov 02 17:56:18.550022 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:56:18.549961    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:56:18.760392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:18.760363    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:18.771460 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:18.771426    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:19.010861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:19.005254    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:56:19.010861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:19.006934    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:56:19.010861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:19.008818    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 17:56:19.901860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:19.888810    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:19.927173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:19.927134    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:56:20.846865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:20.844938    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:26.754450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:26.754413    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 17:56:27.207447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:27.207367    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"]
Nov 02 17:56:27.208750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:27.208716    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy" containerID="cri-o://b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580" gracePeriod=30
Nov 02 17:56:32.186876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.186808    5042 generic.go:356] "Generic (PLEG): container finished" podID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerID="546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8" exitCode=0
Nov 02 17:56:32.187310 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.186892    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" event={"ID":"9cf9580b-aed3-4788-a8f9-fb1427610c9a","Type":"ContainerDied","Data":"546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"}
Nov 02 17:56:32.187375 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.187361    5042 scope.go:122] "RemoveContainer" containerID="546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"
Nov 02 17:56:32.189118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.189089    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:32.189197 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.189161    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:32.189241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.189196    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:32.549551 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.549519    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:56:32.606089 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606057    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.606089 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606096    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.606443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606422    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:32.606699 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606671    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:32.606798 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606748    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.606952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606795    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.606952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.606885    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.608008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.607131    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.608008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.607320    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.608008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.607411    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.608008 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.607538    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data\") pod \"41c8f333-4d05-40fa-a759-543d212b79e2\" (UID: \"41c8f333-4d05-40fa-a759-543d212b79e2\") "
Nov 02 17:56:32.608896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.608815    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41c8f333-4d05-40fa-a759-543d212b79e2-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.608896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.608893    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.609795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.609125    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:32.609795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.609142    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:32.609795 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.609205    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:32.614145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.614095    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:32.614145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.614105    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:56:32.614402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.614369    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:32.621678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.621641    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5" pod "41c8f333-4d05-40fa-a759-543d212b79e2" (UID: "41c8f333-4d05-40fa-a759-543d212b79e2"). InnerVolumeSpecName "kube-api-access-jgdm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729715    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgdm5\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-kube-api-access-jgdm5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729763    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41c8f333-4d05-40fa-a759-543d212b79e2-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729781    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729796    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41c8f333-4d05-40fa-a759-543d212b79e2-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729811    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729824    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:32.737866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:32.729858    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41c8f333-4d05-40fa-a759-543d212b79e2-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:33.218564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.218403    5042 generic.go:356] "Generic (PLEG): container finished" podID="41c8f333-4d05-40fa-a759-543d212b79e2" containerID="b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580" exitCode=0
Nov 02 17:56:33.218564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.218488    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" event={"ID":"41c8f333-4d05-40fa-a759-543d212b79e2","Type":"ContainerDied","Data":"b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580"}
Nov 02 17:56:33.218564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.218530    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds" event={"ID":"41c8f333-4d05-40fa-a759-543d212b79e2","Type":"ContainerDied","Data":"d31f400c4e4b4c4561c1b1b6a6e91b9356d2eb5aee8787a20f73330eb58fdd41"}
Nov 02 17:56:33.218564 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.218552    5042 scope.go:122] "RemoveContainer" containerID="b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580"
Nov 02 17:56:33.219215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.218675    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"
Nov 02 17:56:33.222074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.222043    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" event={"ID":"9cf9580b-aed3-4788-a8f9-fb1427610c9a","Type":"ContainerStarted","Data":"07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea"}
Nov 02 17:56:33.238433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.238405    5042 scope.go:122] "RemoveContainer" containerID="fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"
Nov 02 17:56:33.252634 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.252606    5042 scope.go:122] "RemoveContainer" containerID="b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580"
Nov 02 17:56:33.253077 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:56:33.253043    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580\": container with ID starting with b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580 not found: ID does not exist" containerID="b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580"
Nov 02 17:56:33.253172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.253088    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580"} err="failed to get container status \"b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580\": rpc error: code = NotFound desc = could not find container \"b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580\": container with ID starting with b7259427ef1c4ca976e8f8abd37157cfdd98d35870064f24ac8ff68797397580 not found: ID does not exist"
Nov 02 17:56:33.253172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.253110    5042 scope.go:122] "RemoveContainer" containerID="fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"
Nov 02 17:56:33.254011 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:56:33.253981    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1\": container with ID starting with fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1 not found: ID does not exist" containerID="fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"
Nov 02 17:56:33.254565 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.254018    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1"} err="failed to get container status \"fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1\": rpc error: code = NotFound desc = could not find container \"fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1\": container with ID starting with fae812d94577994f98aa2ad10dece7d21c9af69a5a4b1249f4f1b19489f9b9e1 not found: ID does not exist"
Nov 02 17:56:33.254789 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.254759    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:56:33.255678 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.255647    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:33.255778 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.255690    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:33.309131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.309087    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"]
Nov 02 17:56:33.326919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.318023    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d8e217-openshift-default-7d8b8df87c-npgds"]
Nov 02 17:56:33.567751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.567653    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" path="/var/lib/kubelet/pods/41c8f333-4d05-40fa-a759-543d212b79e2/volumes"
Nov 02 17:56:33.583863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:33.579236    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:56:34.255632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:34.255592    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:34.256183 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:34.255665    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:35.254853 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:35.254813    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:35.255067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:35.254919    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:36.262859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.262694    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:36.262859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.262769    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:36.808926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.806186    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"]
Nov 02 17:56:36.808926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.807224    5042 cpu_manager.go:414] "containerMap: removing container" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy"
Nov 02 17:56:36.808926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.807246    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy"
Nov 02 17:56:36.808926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.807273    5042 cpu_manager.go:414] "containerMap: removing container" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy"
Nov 02 17:56:36.808926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.807281    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy"
Nov 02 17:56:36.808926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.807484    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy"
Nov 02 17:56:36.813948 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.813919    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.942869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.877633    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.957189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.957154    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnk26\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.957529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.957509    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.976006 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.892201    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-7771bf-openshift-default-dockercfg-k8z2z\""
Nov 02 17:56:36.976276 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.926359    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"]
Nov 02 17:56:36.976549 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.976506    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.976772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.976755    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.976998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.976982    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:36.985169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:36.985122    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.041866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.020957    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.041866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.021071    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.124955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.123916    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.124955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.123975    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnk26\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.124955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.124035    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.162071    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.162227    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.162310    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.162406    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.162476    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.162546    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.169292    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.173419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.169565    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.272621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.239337    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.272621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.239744    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.272621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.262860    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:37.272621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.262926    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:37.272621 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.266533    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.321916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.294516    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.321916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.295106    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnk26\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.321916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.306430    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.321916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.319210    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token\") pod \"e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") " pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.435930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.435799    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:37.898620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:37.898574    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"]
Nov 02 17:56:38.083425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.083377    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:38.083710 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.083684    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:38.083884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.083863    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:38.255425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.255390    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:38.255641 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.255449    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:38.343998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.343952    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" event={"ID":"4650ea1b-88c4-4333-b553-27f88d815e90","Type":"ContainerStarted","Data":"c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"}
Nov 02 17:56:38.344551 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.344028    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" event={"ID":"4650ea1b-88c4-4333-b553-27f88d815e90","Type":"ContainerStarted","Data":"adf0d5601c154dac8a6ccdd2c377ef87d079c1c26298110dd3e733479b82b007"}
Nov 02 17:56:38.375869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.374784    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podStartSLOduration=2.374764105 podStartE2EDuration="2.374764105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:38.370331518 +0000 UTC m=+1410.949740910" watchObservedRunningTime="2026-11-02 17:56:38.374764105 +0000 UTC m=+1410.954173493"
Nov 02 17:56:38.438947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:38.436256    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:39.255580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:39.255538    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Nov 02 17:56:39.255913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:39.255611    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Nov 02 17:56:39.443721 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:39.443687    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:40.260032 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:40.260004    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:56:40.261248 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:40.261230    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 17:56:40.366492 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:40.366458    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:40.367441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:40.367420    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:56:44.249157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:44.249116    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"]
Nov 02 17:56:44.250482 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:44.250443    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy" containerID="cri-o://a5648131ef4d40cf6becb7e796645ae88ec4a724de757afdfc2a6507748026ad" gracePeriod=30
Nov 02 17:56:48.740866 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:56:48.737062    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:56:49.500000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.484510    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:56:49.536927 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.519143    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:56:49.569953 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.568938    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:56:49.815184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.815081    5042 generic.go:356] "Generic (PLEG): container finished" podID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerID="a5648131ef4d40cf6becb7e796645ae88ec4a724de757afdfc2a6507748026ad" exitCode=0
Nov 02 17:56:49.815864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.815812    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm" event={"ID":"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6","Type":"ContainerDied","Data":"a5648131ef4d40cf6becb7e796645ae88ec4a724de757afdfc2a6507748026ad"}
Nov 02 17:56:49.816063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.816046    5042 scope.go:122] "RemoveContainer" containerID="846543a8927fd97c3dea5d16376205e81340f4c02cc45eb3d8233061700dceae"
Nov 02 17:56:49.860087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:49.860040    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:50.170624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.170572    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:50.419925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.418880    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:56:50.498133 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.496942    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:50.521874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.512611    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:56:50.739932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.534531    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:56:50.739932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.535214    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:56:50.739932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.576996    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:50.739932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.577643    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:56:50.789867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.765436    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 02 17:56:50.894869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.870036    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:50.894869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:50.888539    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:56:51.223693 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.223406    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:56:51.350921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.314695    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:56:51.350921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.338224    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:56:51.504305 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.494461    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:51.536969 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.536931    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:56:51.608125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.536989    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:51.710862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.706011    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:51.717123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.717040    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:51.812028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.794325    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:51.888726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:51.888613    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:52.046691 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.046659    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154085    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154153    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154248    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154297    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154416    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154501    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154534    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154680    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.155926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.154717    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs\") pod \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\" (UID: \"60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6\") "
Nov 02 17:56:52.172053 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.172005    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:52.180867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.180499    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:52.184307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.182353    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:52.184307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.183256    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"
Nov 02 17:56:52.190881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.184651    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:52.191293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.191264    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:52.208558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.208523    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:56:52.247029 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.224928    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:52.302591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.280743    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:52.363941 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.303050    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.364144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.364125    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.408186 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.408152    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.408380 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.408365    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.408501 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.408487    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.408584 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.363178    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:52.412901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.336679    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:52.413115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.387904    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:52.413242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.408049    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:56:52.456539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.456498    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2" pod "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" (UID: "60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6"). InnerVolumeSpecName "kube-api-access-f7qm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:52.571878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.547411    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qm2\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-kube-api-access-f7qm2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.684074 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.683987    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.684320 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.684301    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.684440 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.684426    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:56:52.684537 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.570124    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:52.684990 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.622939    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:52.718983 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.718921    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:56:52.948921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:52.903936    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:53.026278 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.026208    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"]
Nov 02 17:56:53.411056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.410941    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:53.498308 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.423168    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-6f78de-openshift-default-5bdcbfdfd5-v7cnm"]
Nov 02 17:56:53.608905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.608860    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" path="/var/lib/kubelet/pods/60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6/volumes"
Nov 02 17:56:53.671570 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.670548    5042 scope.go:122] "RemoveContainer" containerID="a5648131ef4d40cf6becb7e796645ae88ec4a724de757afdfc2a6507748026ad"
Nov 02 17:56:53.705734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.689117    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:53.705734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.689558    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:56:53.724879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.724820    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:53.944052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:53.944003    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:54.232013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.231962    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:54.294867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.292666    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:56:54.504928 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.504445    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:56:54.706470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.706418    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:56:54.707146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.707111    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:54.732865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.731897    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:54.996508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:54.968033    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:55.256099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.255982    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:56:55.256624 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.256593    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:56:55.526284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.526175    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:55.849867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.526871    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:56:55.850991 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.724407    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:56:55.864374 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.724473    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:55.865925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.741971    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:55.994868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:55.993883    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:56:56.014694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.014659    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:56:56.268994 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.268938    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:56:56.856293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.855196    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:56:56.867935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.866933    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:56:56.867935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.867236    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:56:56.867935 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.867794    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:56:56.868258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:56.867984    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:56:57.004867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.002045    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:56:57.170407 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.170321    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:56:57.254868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.252598    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:56:57.281169 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.281128    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:56:57.304480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.303797    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:57.457694 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.457061    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:56:57.470108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.470084    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:56:57.471514 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.471485    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:56:57.625041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:57.623239    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:56:58.933655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.933611    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"]
Nov 02 17:56:58.934741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.934704    5042 cpu_manager.go:414] "containerMap: removing container" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy"
Nov 02 17:56:58.934741 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.934731    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy"
Nov 02 17:56:58.934977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.934765    5042 cpu_manager.go:414] "containerMap: removing container" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy"
Nov 02 17:56:58.934977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.934772    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy"
Nov 02 17:56:58.935567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.935363    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy"
Nov 02 17:56:58.935567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.935409    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c8d6f7-64d7-47d4-a5dc-7ccd60650eb6" containerName="istio-proxy"
Nov 02 17:56:58.935567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.935421    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c8f333-4d05-40fa-a759-543d212b79e2" containerName="istio-proxy"
Nov 02 17:56:58.940146 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.940118    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:58.949998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.948788    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-86f5f0\"/\"auth-test-model-86f5f0-kserve-self-signed-certs\""
Nov 02 17:56:58.949998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.949578    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-86f5f0\"/\"kube-root-ca.crt\""
Nov 02 17:56:58.949998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.949682    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-86f5f0\"/\"openshift-service-ca.crt\""
Nov 02 17:56:58.949998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.949822    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-86f5f0\"/\"default-dockercfg-jpdf8\""
Nov 02 17:56:58.978094 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:58.978054    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"]
Nov 02 17:56:59.032203 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.032165    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.035107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.035072    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnjv\" (UniqueName: \"kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.035358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.035339    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.035509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.035495    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.035637 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.035624    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.035767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.035754    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.036653 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.033280    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.053683 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.053639    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"]
Nov 02 17:56:59.054622 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.054595    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:56:59.055072 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.055051    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:56:59.055263 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.055226    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:56:59.055422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.055407    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 17:56:59.104787 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.103436    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"]
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.143895    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144041    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnjv\" (UniqueName: \"kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144081    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144111    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mhsv\" (UniqueName: \"kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144164    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144208    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144263    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144287    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.144348    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.172925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.145122    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.216588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.192664    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.216588 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.192933    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.273750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.258352    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.273750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.259715    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mhsv\" (UniqueName: \"kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.273750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.259851    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.273750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.259913    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.273750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.261891    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.273750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.266912    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnjv\" (UniqueName: \"kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.297535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.297496    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.298251 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.298223    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm\") pod \"auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") " pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.315610 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.315540    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mhsv\" (UniqueName: \"kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv\") pod \"maas-api-revoke-sub-936ac2e1-tcw9p\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.404915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.404509    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:56:59.407649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.407566    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:56:59.997909 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:56:59.997830    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"]
Nov 02 17:57:00.004219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:00.004014    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"]
Nov 02 17:57:00.335422 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:00.335381    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p" event={"ID":"2047d9fb-6cc6-4e1e-885e-7d5732943848","Type":"ContainerStarted","Data":"26435d3589f185e24544f183755ca28af0f107ec272ff280a281a0f13df2aaad"}
Nov 02 17:57:00.337728 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:00.337694    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" event={"ID":"cd74de39-73ad-4ad7-9bac-ce575fb6219d","Type":"ContainerStarted","Data":"60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188"}
Nov 02 17:57:00.337921 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:00.337737    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" event={"ID":"cd74de39-73ad-4ad7-9bac-ce575fb6219d","Type":"ContainerStarted","Data":"d3899e623afffc69405d66dd93fb0c4f94f49b27116e3bafe4028a5ce3a86870"}
Nov 02 17:57:00.363884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:00.361472    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podStartSLOduration=2.361457231 podStartE2EDuration="2.361457231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:00.358252168 +0000 UTC m=+1432.937661556" watchObservedRunningTime="2026-11-02 17:57:00.361457231 +0000 UTC m=+1432.940866639"
Nov 02 17:57:00.409877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:00.408578    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:01.350746 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:01.350657    5042 generic.go:356] "Generic (PLEG): container finished" podID="2047d9fb-6cc6-4e1e-885e-7d5732943848" containerID="bae7bdc271ec45700ee965b3a60594f9ecb9136ecb3ce7d02dd7cdfe85fff0a5" exitCode=0
Nov 02 17:57:01.351246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:01.350749    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p" event={"ID":"2047d9fb-6cc6-4e1e-885e-7d5732943848","Type":"ContainerDied","Data":"bae7bdc271ec45700ee965b3a60594f9ecb9136ecb3ce7d02dd7cdfe85fff0a5"}
Nov 02 17:57:03.050502 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.050469    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:57:03.156924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.156875    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca\") pod \"2047d9fb-6cc6-4e1e-885e-7d5732943848\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") "
Nov 02 17:57:03.157126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.156942    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv\" (UniqueName: \"kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv\") pod \"2047d9fb-6cc6-4e1e-885e-7d5732943848\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") "
Nov 02 17:57:03.157126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.157025    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca\") pod \"2047d9fb-6cc6-4e1e-885e-7d5732943848\" (UID: \"2047d9fb-6cc6-4e1e-885e-7d5732943848\") "
Nov 02 17:57:03.157302 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.157276    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca" pod "2047d9fb-6cc6-4e1e-885e-7d5732943848" (UID: "2047d9fb-6cc6-4e1e-885e-7d5732943848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:03.157726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.157657    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:03.159373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.159349    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca" pod "2047d9fb-6cc6-4e1e-885e-7d5732943848" (UID: "2047d9fb-6cc6-4e1e-885e-7d5732943848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:57:03.159558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.159542    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv" pod "2047d9fb-6cc6-4e1e-885e-7d5732943848" (UID: "2047d9fb-6cc6-4e1e-885e-7d5732943848"). InnerVolumeSpecName "kube-api-access-6mhsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:03.259340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.259302    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mhsv\" (UniqueName: \"kubernetes.io/projected/2047d9fb-6cc6-4e1e-885e-7d5732943848-kube-api-access-6mhsv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:03.259340 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.259338    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2047d9fb-6cc6-4e1e-885e-7d5732943848-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:03.374898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.374571    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"
Nov 02 17:57:03.374898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.374604    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p" event={"ID":"2047d9fb-6cc6-4e1e-885e-7d5732943848","Type":"ContainerDied","Data":"26435d3589f185e24544f183755ca28af0f107ec272ff280a281a0f13df2aaad"}
Nov 02 17:57:03.374898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:03.374704    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26435d3589f185e24544f183755ca28af0f107ec272ff280a281a0f13df2aaad"
Nov 02 17:57:05.108056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:05.108012    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:05.128090 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:05.108084    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:05.938866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:05.927035    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:05.938866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:05.927105    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:07.664801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.664330    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"]
Nov 02 17:57:07.667915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.667552    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2047d9fb-6cc6-4e1e-885e-7d5732943848" containerName="revoke-keys"
Nov 02 17:57:07.667915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.667577    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2047d9fb-6cc6-4e1e-885e-7d5732943848" containerName="revoke-keys"
Nov 02 17:57:07.667915 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.667759    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2047d9fb-6cc6-4e1e-885e-7d5732943848" containerName="revoke-keys"
Nov 02 17:57:07.703573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.702500    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"]
Nov 02 17:57:07.703573 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.702623    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.772000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.735214    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-23a657-openshift-default-dockercfg-2n62r\""
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950310    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950395    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950441    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950519    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950601    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950663    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950716    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950741    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:07.962671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:07.950764    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lljc\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073288 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073227    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073288 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073284    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lljc\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073357    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073450    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073521 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073502    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073597    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073738 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073702    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073820 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073786    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.073892 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.073872    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.074352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.074324    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.075157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.075124    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.075261 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.075200    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.076145 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.076110    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.076613 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.076570    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.080473 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.080446    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.085786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.085512    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.089669 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.089616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lljc\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.090213 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.090163    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token\") pod \"e2e-explicit-23a657-openshift-default-7db6744997-ctqqs\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") " pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.336583 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.336486    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:08.733852 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.733781    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"]
Nov 02 17:57:08.742025 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:57:08.741988    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f72c7c2_6a6d_4c93_aedb_eb1a5fd19597.slice/crio-eecc427a5f5fb72681e008cafaeee69cb8ad4fe66b737e153eaeb97dd9cd092b WatchSource:0}: Error finding container eecc427a5f5fb72681e008cafaeee69cb8ad4fe66b737e153eaeb97dd9cd092b: Status 404 returned error can't find the container with id eecc427a5f5fb72681e008cafaeee69cb8ad4fe66b737e153eaeb97dd9cd092b
Nov 02 17:57:08.745879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.745796    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:08.746001 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.745974    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:08.746067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.746019    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:08.749910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.749593    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"]
Nov 02 17:57:08.841904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.830334    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"]
Nov 02 17:57:08.841904 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.830467    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.928928    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929130    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929202    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929244    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929333    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929389    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929415    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929466    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:08.931307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:08.929501    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlls\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036245    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlls\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036369    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036522    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036583    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036620    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036700    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.072825 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.036750    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.081109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.076053    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.081109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.076675    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.081109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.078702    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.081109 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.079356    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.082421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.081884    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.082421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.081970    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.082421 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.082374    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.118020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.117983    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.119791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.119726    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlls\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.120337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.120284    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.125389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.125327    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.249931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.249623    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:09.405989 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.405950    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:09.429745 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.429670    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:09.499219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.499106    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" event={"ID":"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597","Type":"ContainerStarted","Data":"2ac7909c9d0d111548febe3398dd73cc1d8522a97f0d2567c46e3452caa0dfab"}
Nov 02 17:57:09.499219 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.499171    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" event={"ID":"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597","Type":"ContainerStarted","Data":"eecc427a5f5fb72681e008cafaeee69cb8ad4fe66b737e153eaeb97dd9cd092b"}
Nov 02 17:57:09.533163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.531601    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podStartSLOduration=2.531583413 podStartE2EDuration="2.531583413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:09.530677233 +0000 UTC m=+1442.110086619" watchObservedRunningTime="2026-11-02 17:57:09.531583413 +0000 UTC m=+1442.110992804"
Nov 02 17:57:09.534236 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.533889    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:09.764700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.744427    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:09.764700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.744526    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:09.764700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.744572    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:09.785925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:09.773960    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"]
Nov 02 17:57:10.339388 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.339275    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:10.346459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.346100    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:10.515572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.515192    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" event={"ID":"cca681ab-7126-4c85-9246-38815d12500f","Type":"ContainerStarted","Data":"d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a"}
Nov 02 17:57:10.515572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.515254    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" event={"ID":"cca681ab-7126-4c85-9246-38815d12500f","Type":"ContainerStarted","Data":"5fc572af64570fc929d6903ec81d229376784aa214f41bc66e1f1e7e0e3e9dbb"}
Nov 02 17:57:10.518781 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.517807    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:10.521163 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.521105    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:57:10.553242 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:10.553177    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podStartSLOduration=2.553155988 podStartE2EDuration="2.553155988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:10.543596847 +0000 UTC m=+1443.123006234" watchObservedRunningTime="2026-11-02 17:57:10.553155988 +0000 UTC m=+1443.132565375"
Nov 02 17:57:11.250968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.250821    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:11.252965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.252637    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:11.252965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.252697    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:11.258043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.257951    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:11.258043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.257996    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:11.703420 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.701996    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:11.703420 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:11.702090    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:12.256901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:12.250888    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:12.256901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:12.250952    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:13.251539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.251465    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:13.251539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.251534    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:13.458361 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.458314    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"]
Nov 02 17:57:13.558916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.541934    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"]
Nov 02 17:57:13.558916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.545342    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:13.612987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.581872    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-23a657\""
Nov 02 17:57:13.612987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.581989    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-23a657-dockercfg-2975k\""
Nov 02 17:57:13.748359 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.748320    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:13.748620 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.748599    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zfp\" (UniqueName: \"kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp\") pod \"payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:13.875878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.875360    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:13.875878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.875421    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zfp\" (UniqueName: \"kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp\") pod \"payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:13.879147 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.878377    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:13.890929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.888426    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"]
Nov 02 17:57:13.944459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.919085    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"]
Nov 02 17:57:13.944459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.919227    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:13.967189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:13.966966    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zfp\" (UniqueName: \"kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp\") pod \"payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:14.037069 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.035169    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:14.128441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.128113    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrl6\" (UniqueName: \"kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6\") pod \"payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") " pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.128441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.128176    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") " pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.231354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.230622    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrl6\" (UniqueName: \"kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6\") pod \"payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") " pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.231354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.230680    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") " pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.232544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.232167    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume\") pod \"payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") " pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.253799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.250601    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrl6\" (UniqueName: \"kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6\") pod \"payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") " pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.253799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.251410    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:14.253799 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.251450    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:14.365185 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.364147    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:14.636377 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.636340    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"]
Nov 02 17:57:14.910630 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:57:14.910549    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fbdf07_890f_45a5_8764_c7f4b901b1f6.slice/crio-585800ec1b775264cecf3321b4c2692f53c843ff2dc49cb44abf522572f2caec WatchSource:0}: Error finding container 585800ec1b775264cecf3321b4c2692f53c843ff2dc49cb44abf522572f2caec: Status 404 returned error can't find the container with id 585800ec1b775264cecf3321b4c2692f53c843ff2dc49cb44abf522572f2caec
Nov 02 17:57:14.920442 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:14.920348    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"]
Nov 02 17:57:15.052224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.052190    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" event={"ID":"feef2c75-54bb-4ac1-868e-b17a58cf5275","Type":"ContainerStarted","Data":"8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626"}
Nov 02 17:57:15.052404 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.052247    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" event={"ID":"feef2c75-54bb-4ac1-868e-b17a58cf5275","Type":"ContainerStarted","Data":"ab43c47dfb17c0a3c87ae6f52ecaa09faf3809f0dcb28004c3c0b74b0cee07f5"}
Nov 02 17:57:15.053259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.053239    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:15.055897 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.054868    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8" event={"ID":"f2fbdf07-890f-45a5-8764-c7f4b901b1f6","Type":"ContainerStarted","Data":"585800ec1b775264cecf3321b4c2692f53c843ff2dc49cb44abf522572f2caec"}
Nov 02 17:57:15.084723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.084469    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" podStartSLOduration=2.084446653 podStartE2EDuration="2.084446653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:15.072018624 +0000 UTC m=+1447.651428011" watchObservedRunningTime="2026-11-02 17:57:15.084446653 +0000 UTC m=+1447.663856041"
Nov 02 17:57:15.251864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.251771    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:15.252411 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.252114    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:15.941714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.941662    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"]
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.953406    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:15.957412    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"]
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.062697    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.062778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.062827    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.062900    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.062999    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qlx6\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.063028    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.063062    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.063085    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.063109    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.110905    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8" event={"ID":"f2fbdf07-890f-45a5-8764-c7f4b901b1f6","Type":"ContainerStarted","Data":"ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c"}
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.112186    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.146818    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8" podStartSLOduration=3.146795862 podStartE2EDuration="3.146795862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:16.140215193 +0000 UTC m=+1448.719624580" watchObservedRunningTime="2026-11-02 17:57:16.146795862 +0000 UTC m=+1448.726205248"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164442    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qlx6\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164477    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164502    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164517    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164534    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164686    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164730    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164762    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.164807    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.165829    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.166039    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.166189    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.167181    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.167230    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.169788    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.170535    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.213591    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.215424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.215027    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qlx6\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.251890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.251272    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:16.251890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.251323    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:16.253672 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.253354    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"]
Nov 02 17:57:16.364954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.308420    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"]
Nov 02 17:57:16.364954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.308621    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.364954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.324042    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.427891    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.427958    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.427989    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6fzf\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.428018    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.428062    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.428116    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.428149    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.428402    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.428767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.428461    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.529716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.529680    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.529883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.529752    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.529883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.529870    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.530022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.529917    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.530022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.529946    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6fzf\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.530022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.529974    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.530022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.530018    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.530241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.530073    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.530241 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.530106    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.531947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.530796    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.531947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.531071    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.531947 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.531612    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.532966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.532940    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.533603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.533576    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.535757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.534620    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.551099 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.551067    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.551608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.551582    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.552410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.552358    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6fzf\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf\") pod \"e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") " pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.574529 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:16.574489    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a5e54c_3842_4947_859a_a4d1780467c6.slice/crio-conmon-999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 17:57:16.837165 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.837091    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:16.870766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.869046    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"]
Nov 02 17:57:16.870766 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:57:16.869209    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d370441_564c_40be_8e46_a89c086bdd05.slice/crio-c7c7302e6cb32d74b1a44b4d46173f00701593f7871eeaafe13441e3b5f866c2 WatchSource:0}: Error finding container c7c7302e6cb32d74b1a44b4d46173f00701593f7871eeaafe13441e3b5f866c2: Status 404 returned error can't find the container with id c7c7302e6cb32d74b1a44b4d46173f00701593f7871eeaafe13441e3b5f866c2
Nov 02 17:57:16.923796 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.923714    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:16.924010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.923866    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:16.924010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:16.923912    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:17.122602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.121965    5042 generic.go:356] "Generic (PLEG): container finished" podID="61a5e54c-3842-4947-859a-a4d1780467c6" containerID="999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e" exitCode=0
Nov 02 17:57:17.122602 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.122079    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" event={"ID":"61a5e54c-3842-4947-859a-a4d1780467c6","Type":"ContainerDied","Data":"999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"}
Nov 02 17:57:17.124580 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.123778    5042 scope.go:122] "RemoveContainer" containerID="999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"
Nov 02 17:57:17.127284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.126621    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:17.127284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.126717    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:17.127284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.126761    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:17.128812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.128072    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" event={"ID":"4d370441-564c-40be-8e46-a89c086bdd05","Type":"ContainerStarted","Data":"c7c7302e6cb32d74b1a44b4d46173f00701593f7871eeaafe13441e3b5f866c2"}
Nov 02 17:57:17.333402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.329659    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:57:17.333402 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.329720    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 02 17:57:17.354303 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:57:17.354249    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3249565_04a0_496f_92c1_afe90b3c5c90.slice/crio-de5b6d2d34a871f360809288dd9b5921c1ded09f29665b453124d060d6d66e22 WatchSource:0}: Error finding container de5b6d2d34a871f360809288dd9b5921c1ded09f29665b453124d060d6d66e22: Status 404 returned error can't find the container with id de5b6d2d34a871f360809288dd9b5921c1ded09f29665b453124d060d6d66e22
Nov 02 17:57:17.374416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.374336    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"]
Nov 02 17:57:17.419924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.419314    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:17.419924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.419413    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:17.419924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:17.419459    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:18.145973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.145931    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" event={"ID":"61a5e54c-3842-4947-859a-a4d1780467c6","Type":"ContainerStarted","Data":"5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530"}
Nov 02 17:57:18.149616 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.149247    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" event={"ID":"4d370441-564c-40be-8e46-a89c086bdd05","Type":"ContainerStarted","Data":"9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74"}
Nov 02 17:57:18.152293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.152255    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" event={"ID":"a3249565-04a0-496f-92c1-afe90b3c5c90","Type":"ContainerStarted","Data":"d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6"}
Nov 02 17:57:18.152446 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.152323    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" event={"ID":"a3249565-04a0-496f-92c1-afe90b3c5c90","Type":"ContainerStarted","Data":"de5b6d2d34a871f360809288dd9b5921c1ded09f29665b453124d060d6d66e22"}
Nov 02 17:57:18.207543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.206096    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podStartSLOduration=3.206072969 podStartE2EDuration="3.206072969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:18.200442633 +0000 UTC m=+1450.779852019" watchObservedRunningTime="2026-11-02 17:57:18.206072969 +0000 UTC m=+1450.785482357"
Nov 02 17:57:18.243269 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.237802    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podStartSLOduration=2.237780511 podStartE2EDuration="2.237780511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:18.233228751 +0000 UTC m=+1450.812638138" watchObservedRunningTime="2026-11-02 17:57:18.237780511 +0000 UTC m=+1450.817189898"
Nov 02 17:57:18.304726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.303557    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:18.304726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.304617    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:18.324859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.318003    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 17:57:18.348861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.346380    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:18.348861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.348039    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:18.348861 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.348097    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:18.766390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.761914    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:57:18.766390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.762064    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:18.766390 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.762109    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:18.813831 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:18.807054    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:57:18.865764 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.865699    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:19.018922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.899713    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:19.018922 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:18.899775    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:19.364925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.332668    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:19.365720 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.365684    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:19.864883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.763553    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:19.864883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.763616    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:19.864883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.862938    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:19.864883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.863019    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:19.868658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:19.868618    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:20.064911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.048141    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:57:20.256144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.252497    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:20.256144 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.252596    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:20.350862 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.346544    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:57:20.367013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.364530    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:20.367013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.364598    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.824293    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.824589    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.825502    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.825540    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.848264    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.848331    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.900752    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:20.958865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:20.901163    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:57:21.015785 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.015746    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.339649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.339494    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:21.339937 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.339905    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:21.582901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.574931    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.763552 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.763515    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:21.763898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.763864    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:21.847640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.843510    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:21.847640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.843572    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:21.847640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.845661    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.847640 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.846024    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:57:21.862880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.862660    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.894864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.888996    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.894864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.890107    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:57:21.907229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.906488    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.907229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.906714    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:21.907229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:21.906932    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:57:22.220576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.218628    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:22.331371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.331335    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:22.331651 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.331618    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:22.600923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.600619    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded"
Nov 02 17:57:22.770618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.770568    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:22.770929 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.770894    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:22.844866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.841649    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:22.844866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.841712    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:22.846332 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.846021    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:22.882924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.872294    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:22.905087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.887142    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:57:22.905087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.890381    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:22.908860 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:22.907956    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:23.162742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.148658    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"]
Nov 02 17:57:23.290560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.289923    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:23.333679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.327245    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:23.333679 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.327309    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:23.434474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.434387    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"]
Nov 02 17:57:23.434806 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.434787    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:57:23.435054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.435038    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636290    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636483    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636523    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636614    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636716    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636750    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.636797    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:24.227931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:23.650455    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:24.582474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.576404    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:57:24.741246 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.741186    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:24.745626 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.745585    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f" podUID="9f804eed-b3bb-4531-81b5-6060fd90682b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded"
Nov 02 17:57:24.747092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.747063    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded"
Nov 02 17:57:24.854052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.833421    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:24.854052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.833500    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:24.905957 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.905113    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:57:24.987949 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.954190    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:57:24.995339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.995289    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:24.995698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:24.995654    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:25.035066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.035021    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.035389 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.035369    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.277865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.274242    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:25.281992 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.281938    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:25.437872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.437413    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:25.437872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.437491    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:25.518865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.518001    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:25.518865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.518114    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:25.551086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.550956    5042 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-622zn container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.16:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:25.551411 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.551379    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-622zn" podUID="17ccbff4-2818-4e70-b2f4-a64c92f789e8" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:25.583100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.583035    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4pb5f" podUID="9f804eed-b3bb-4531-81b5-6060fd90682b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:25.652934 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:25.617774    5042 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.039s"
Nov 02 17:57:25.696281 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.696236    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.699971    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.619022    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744194    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744280    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744429    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744619    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744659    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744764    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.744927    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.631918    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.797286 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.777141    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:25.985868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.982378    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:25.987273 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.987235    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:25.987611 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:25.987585    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": context deadline exceeded"
Nov 02 17:57:26.015335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.015263    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:26.017882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.017819    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:26.047762 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.047726    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:26.073107 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.073062    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:26.119207 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.109708    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.180980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.144296    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.291716 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.272006    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.321980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.276992    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:26.321980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.317237    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:26.321980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.277621    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.321980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.289230    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:26.321980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.317399    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:26.446865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.424550    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.571323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.562062    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 02 17:57:26.571323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.562135    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 02 17:57:26.608102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.592665    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:26.608102 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.592745    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:26.643005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.641944    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.812548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.659615    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:26.812548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.679016    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:26.812548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.679092    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:26.812548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.717153    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:26.812548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.717904    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:57:26.812548 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.718872    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:57:26.847930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.820569    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:57:26.847930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.820628    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 02 17:57:26.847930 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:26.844504    5042 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.191s"
Nov 02 17:57:26.847930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.844626    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:26.847930 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.844674    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"]
Nov 02 17:57:26.889881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.886874    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:26.889881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.886939    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.985268    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.996974    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:26.997966    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.021954    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.055104    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.055204    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.055231    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"]
Nov 02 17:57:27.060870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.056636    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.061453 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.060995    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:27.063986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.061638    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:57:27.399151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.341566    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:27.461429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.461400    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.467632    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtsd\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.467928    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468130    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468160    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468370    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468430    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468516    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468624    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.485576 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.468662    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.501903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.486475    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:57:27.602441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.601961    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.650921    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.650972    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651229    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651279    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651348    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651414    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651449    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651507    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtsd\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.651700    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.652867    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.658783    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.659503    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.674917 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.674857    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.742858 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.742144    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.760647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.756759    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.806863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.779313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtsd\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.832988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.832955    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:27.833284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.833266    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:57:27.833418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.833408    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:27.833545 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.833535    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:57:27.833655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.833647    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:57:27.833757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.833748    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:57:27.833880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.833870    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.948168    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.948222    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.948272    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.868768    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.948555    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token\") pod \"e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") " pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.870271    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body=
Nov 02 17:57:27.953267 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.948637    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused"
Nov 02 17:57:27.959152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:27.959070    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 17:57:28.271056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.269428    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:28.314344 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.314317    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:57:28.464429 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.464391    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"]
Nov 02 17:57:28.531337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.529970    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:28.531337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.530077    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:28.531337 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.530124    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:28.825028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.819203    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" event={"ID":"9d557769-1af8-496c-80a5-c5bccf34f277","Type":"ContainerStarted","Data":"56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3"}
Nov 02 17:57:28.825028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.819299    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" event={"ID":"9d557769-1af8-496c-80a5-c5bccf34f277","Type":"ContainerStarted","Data":"ec04e63777926c709d02fdef6e93d60b7d69d961e1c657d7f751f42838bb995b"}
Nov 02 17:57:28.850455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.849325    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:28.850455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.850203    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:28.851171 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.851103    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 17:57:28.874876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:28.864582    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podStartSLOduration=6.864562143 podStartE2EDuration="6.864562143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:28.857272294 +0000 UTC m=+1461.436681682" watchObservedRunningTime="2026-11-02 17:57:28.864562143 +0000 UTC m=+1461.443971531"
Nov 02 17:57:29.075182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.074894    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.075182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.075004    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.075182 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.075051    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.239598 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.239566    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"]
Nov 02 17:57:29.484140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.479422    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:29.484140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.483324    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:29.484140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.483382    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:29.847751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.845039    5042 generic.go:356] "Generic (PLEG): container finished" podID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerID="3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb" exitCode=0
Nov 02 17:57:29.847751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.845127    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" event={"ID":"a7f975ec-36ec-401d-b5a8-cca7fcc299fb","Type":"ContainerDied","Data":"3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"}
Nov 02 17:57:29.847751 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.845647    5042 scope.go:122] "RemoveContainer" containerID="3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"
Nov 02 17:57:29.851176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.851098    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" event={"ID":"038c55b6-2933-42e9-a931-3e9c9e6210c0","Type":"ContainerStarted","Data":"70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c"}
Nov 02 17:57:29.851176 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.851172    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" event={"ID":"038c55b6-2933-42e9-a931-3e9c9e6210c0","Type":"ContainerStarted","Data":"998e7bf78aeac41128a46d96120af2bc30bbfc4fa3384756814b0b914717cab4"}
Nov 02 17:57:29.854266 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.854234    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.854458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.854443    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.854556 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.854545    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.997867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:29.985234    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podStartSLOduration=6.985210174 podStartE2EDuration="6.985210174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:29.982473837 +0000 UTC m=+1462.561883224" watchObservedRunningTime="2026-11-02 17:57:29.985210174 +0000 UTC m=+1462.564619561"
Nov 02 17:57:30.434828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:30.294626    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:30.434828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:30.296655    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:30.434828 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:30.296707    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:30.488198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:30.470143    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:30.488198 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:30.470209    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:31.051148 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.051056    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" event={"ID":"a7f975ec-36ec-401d-b5a8-cca7fcc299fb","Type":"ContainerStarted","Data":"1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726"}
Nov 02 17:57:31.291036 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.290995    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:31.291259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.291063    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:31.469932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.469896    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:31.470121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.469952    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:31.733091 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.732981    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:57:31.733866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.733816    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:31.734076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:31.734047    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:32.127931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.114670    5042 generic.go:356] "Generic (PLEG): container finished" podID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470" exitCode=0
Nov 02 17:57:32.127931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.114906    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" event={"ID":"db2d4865-b81a-4d68-9dde-1bc29f58bde7","Type":"ContainerDied","Data":"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"}
Nov 02 17:57:32.127931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.115476    5042 scope.go:122] "RemoveContainer" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"
Nov 02 17:57:32.130903 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.130857    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:32.131181 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.131155    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:32.131354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.131334    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:32.294864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.294400    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:32.294864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.294460    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:32.395988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.382401    5042 generic.go:356] "Generic (PLEG): container finished" podID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerID="d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19" exitCode=0
Nov 02 17:57:32.395988 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.383476    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" event={"ID":"4a3aadd4-5c54-4a81-ab70-8258d5e042ee","Type":"ContainerDied","Data":"d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"}
Nov 02 17:57:32.428866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.420916    5042 scope.go:122] "RemoveContainer" containerID="d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"
Nov 02 17:57:32.428866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.427333    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:32.428866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.427470    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:32.428866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.427520    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:32.504888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.500629    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:32.504888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.500688    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:32.802671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.771975    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:32.802671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.773005    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:32.923868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.899392    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:32.923868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:32.912994    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:57:33.004868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.004016    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:33.207962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.068496    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:33.207962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.068556    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:33.485882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.436694    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:33.485882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.436750    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:33.631086 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.631047    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:33.631370 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.631341    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:33.680011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.679963    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:33.968127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.968067    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:57:34.333585 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:33.977264    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:57:34.356178 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.017719    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:34.356470 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.356437    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:34.578899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.074789    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:57:34.578899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.578611    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:57:34.579905 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.087147    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:34.606360 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.018385    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:57:34.608283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.484941    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:34.608535 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.608509    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:34.608658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.608361    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:34.608857 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.608825    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:57:34.609245 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.533713    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:34.609472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.539464    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:57:34.610227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.510519    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:34.610464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.610441    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:34.611134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.611115    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:57:34.767264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.767173    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:34.767532 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.767505    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:34.790876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.776682    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:57:34.790876 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:34.784282    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:57:35.327247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.315147    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:57:35.471726 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.407924    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:35.472048 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.472016    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:35.472200 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.408999    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:57:35.472899 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.408939    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:57:35.498869 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.473681    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:57:35.742124 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.530039    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:35.745980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.745939    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:35.755040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.624145    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:35.806405 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.624087    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:57:35.806987 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.713005    5042 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-vktlz container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:57:35.807223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.807175    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz" podUID="fa18f4d4-d9fa-4802-b90b-e7044d222b39" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:35.807368 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.742081    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:35.807496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.807476    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:35.808188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:35.624021    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:36.242011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.237592    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:36.471691 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.471634    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:36.476142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.476088    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:36.502870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.476459    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:36.545403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.505776    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:36.545403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.524185    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:36.545403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.524242    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:36.545403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.525758    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:36.707573 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:36.707533    5042 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.076s"
Nov 02 17:57:36.707898 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.707877    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:57:36.708125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.708111    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:57:36.740936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.737021    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:36.740936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.737085    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:36.740936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.739435    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" event={"ID":"db2d4865-b81a-4d68-9dde-1bc29f58bde7","Type":"ContainerStarted","Data":"4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58"}
Nov 02 17:57:36.768883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.761873    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:36.772803 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:36.772766    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" event={"ID":"4a3aadd4-5c54-4a81-ab70-8258d5e042ee","Type":"ContainerStarted","Data":"9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92"}
Nov 02 17:57:37.085391 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.085291    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:37.231942 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.223046    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:37.257130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.254151    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:37.257461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.257416    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:37.257641 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.255460    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:57:37.258978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.258942    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:37.268433 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.255395    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:37.270078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.233669    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:57:37.270447 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.270408    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:57:37.357925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.329531    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:37.357925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.329597    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:37.497023 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.496981    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 02 17:57:37.497708 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.497654    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 02 17:57:37.497980 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.497959    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:57:37.499038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.498998    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 17:57:37.590591 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.590525    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:37.727554 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.727521    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:37.728225 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.728194    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:37.766051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.766003    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:37.926802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.912008    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:37.941794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:37.941749    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:38.099961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.099859    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 17:57:38.201098 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.201064    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:38.202772 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.201373    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:38.203054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.202028    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:38.203215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.203192    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:38.242067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.242027    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"]
Nov 02 17:57:38.256142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.256104    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.258101 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.258060    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:38.272115 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.272072    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:38.273295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.273271    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:57:38.275436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.275398    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b1a788\"/\"kube-root-ca.crt\""
Nov 02 17:57:38.290481 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.278626    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-b1a788\"/\"openshift-service-ca.crt\""
Nov 02 17:57:38.290715 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.281608    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body=
Nov 02 17:57:38.290867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.282272    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b1a788\"/\"default-dockercfg-7grnv\""
Nov 02 17:57:38.290968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.282376    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-b1a788\"/\"auth-test-model-b1a788-kserve-self-signed-certs\""
Nov 02 17:57:38.291075 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.286047    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"]
Nov 02 17:57:38.291325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.291297    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused"
Nov 02 17:57:38.561882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.561147    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:38.607434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.607402    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"]
Nov 02 17:57:38.627103 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.607696    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:57:38.627711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.608786    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.668419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.668373    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 17:57:38.680559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.680520    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.681071 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.681028    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.681661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.681348    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.681940 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.681922    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.682155 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.682133    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc2jv\" (UniqueName: \"kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.753866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.741787    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1e96f4b-62scq"]
Nov 02 17:57:38.753866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.741936    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:38.753866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.743629    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:38.754910 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.754881    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 17:57:38.756293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.756269    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:57:38.756474 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.756449    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Nov 02 17:57:38.773914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.773873    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:38.807439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.794146    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.807439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.806253    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc2jv\" (UniqueName: \"kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.807439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.806425    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.807439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.806614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.807439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.806805    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.807439 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.806879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.824943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.815618    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.825212 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.794590    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.825779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.794954    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:38.826437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.826404    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:38.947928 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.934461    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:57:38.994821 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:38.952495    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:57:39.145925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.107098    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:39.145925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.109923    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc2jv\" (UniqueName: \"kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:39.180919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.155210    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm\") pod \"auth-test-model-b1a788-kserve-57d789d69-sfk5c\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") " pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:39.200887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.197258    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:39.200887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.197315    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:39.200887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.197983    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:39.200887 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.198022    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:39.207901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.201513    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:39.221496 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.221143    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:39.288076 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.270652    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:39.313881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.306946    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 17:57:39.532632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.532573    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Nov 02 17:57:39.681373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.677979    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 17:57:39.698698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.698665    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:39.724965 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.724915    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:39.725409 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.725387    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 17:57:39.728671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.728649    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 17:57:39.729727 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.729688    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:39.863916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:39.856947    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:57:40.238797 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.238758    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:40.239164 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.239133    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:40.242088 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242052    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb3bd88-f670-4767-9c08-0a38995e3679" path="/var/lib/kubelet/pods/bcb3bd88-f670-4767-9c08-0a38995e3679/volumes"
Nov 02 17:57:40.242658 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242632    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:57:40.242813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242718    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 17:57:40.242813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242755    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 17:57:40.242813 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242805    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 17:57:40.243040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242857    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 17:57:40.243040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242893    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 17:57:40.243040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242910    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:57:40.243040 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.242952    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 17:57:40.251011 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.250497    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 17:57:40.279240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.268382    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:40.279240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.268452    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:40.394908 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.288714    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 17:57:40.664021 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:40.661306    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:57:40.667499 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:57:40.667264    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e28883_4069_480e_b9c4_58529708f5da.slice/crio-692d89444255941fe21aded4cbe20195c2960d18bab7a19853d11cd72a4232b9 WatchSource:0}: Error finding container 692d89444255941fe21aded4cbe20195c2960d18bab7a19853d11cd72a4232b9: Status 404 returned error can't find the container with id 692d89444255941fe21aded4cbe20195c2960d18bab7a19853d11cd72a4232b9
Nov 02 17:57:41.040995 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.040539    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"]
Nov 02 17:57:41.194034 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.193984    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:41.194312 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.194285    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:41.212881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.212671    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:41.212881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.212733    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:41.446070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.445992    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" event={"ID":"d9e28883-4069-480e-b9c4-58529708f5da","Type":"ContainerStarted","Data":"c3a59077fdc42cf3fab05d77516f57e78c9a4a2c360298b2cd0e7e283d077192"}
Nov 02 17:57:41.446255 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.446242    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" event={"ID":"d9e28883-4069-480e-b9c4-58529708f5da","Type":"ContainerStarted","Data":"692d89444255941fe21aded4cbe20195c2960d18bab7a19853d11cd72a4232b9"}
Nov 02 17:57:41.467723 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:41.467673    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" podStartSLOduration=6.467657673 podStartE2EDuration="6.467657673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:41.463084292 +0000 UTC m=+1474.042493679" watchObservedRunningTime="2026-11-02 17:57:41.467657673 +0000 UTC m=+1474.047067104"
Nov 02 17:57:42.190339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:42.190293    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:42.191070 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:42.191035    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:42.199560 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:42.199526    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:42.199824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:42.199790    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:42.225542 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:42.225503    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:43.200865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:43.195526    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:43.200865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:43.195622    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:43.212131 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:43.212095    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 02 17:57:43.212443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:43.212400    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 02 17:57:44.188946 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:44.188891    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:57:44.189173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:44.188961    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 02 17:57:44.240867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:44.239366    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:57:44.243218 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:44.242074    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 17:57:45.193515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:45.193465    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:57:45.194742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:45.194717    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 17:57:48.542354 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:48.542271    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:57:49.227287 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:49.227232    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:49.270820 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:49.270782    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:49.548259 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:49.548180    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 17:57:53.563323 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.563271    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"]
Nov 02 17:57:53.569816 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.569790    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.575307 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.575278    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"auth-test-model-7771bf-kserve-self-signed-certs\""
Nov 02 17:57:53.578129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.578103    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"]
Nov 02 17:57:53.580283 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.580259    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"kube-root-ca.crt\""
Nov 02 17:57:53.580437 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.580353    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"openshift-service-ca.crt\""
Nov 02 17:57:53.594418 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.594385    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-7771bf\"/\"default-dockercfg-fl2z6\""
Nov 02 17:57:53.623865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.623808    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.624078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.623957    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.624078 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.623991    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kx4\" (UniqueName: \"kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.624205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.624120    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.624386 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.624369    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.624459 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.624429    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.725477 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725429    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.725681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725498    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.725681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725556    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.725681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725612    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.725681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725635    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kx4\" (UniqueName: \"kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.725681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725680    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.726007 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725980    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.726082 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.725993    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.726082 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.726066    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.727975 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.727955    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.728476 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.728450    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.736509 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.736471    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"]
Nov 02 17:57:53.736728 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.736699    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy" containerID="cri-o://b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40" gracePeriod=30
Nov 02 17:57:53.742073 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.742041    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kx4\" (UniqueName: \"kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4\") pod \"auth-test-model-7771bf-kserve-db466b9b5-7kzzc\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") " pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:53.908916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:53.901617    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:54.342293 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:57:54.342259    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fb03c20_d3d4_44a6_a7e4_dfe3139e2b37.slice/crio-37987a39c2941bec6622e1e2e295c2f80426b47aa03bce8545d648a7ad9a9af7 WatchSource:0}: Error finding container 37987a39c2941bec6622e1e2e295c2f80426b47aa03bce8545d648a7ad9a9af7: Status 404 returned error can't find the container with id 37987a39c2941bec6622e1e2e295c2f80426b47aa03bce8545d648a7ad9a9af7
Nov 02 17:57:54.344139 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:54.344117    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"]
Nov 02 17:57:54.584222 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:54.584178    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" event={"ID":"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37","Type":"ContainerStarted","Data":"a9dc045c8474fbcc5ffd55e9106ab3f60fa006aa35e646ba5057e24f901ee836"}
Nov 02 17:57:54.584649 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:54.584233    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" event={"ID":"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37","Type":"ContainerStarted","Data":"37987a39c2941bec6622e1e2e295c2f80426b47aa03bce8545d648a7ad9a9af7"}
Nov 02 17:57:54.604933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:54.604813    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" podStartSLOduration=1.604799589 podStartE2EDuration="1.604799589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:54.602040003 +0000 UTC m=+1487.181449391" watchObservedRunningTime="2026-11-02 17:57:54.604799589 +0000 UTC m=+1487.184209023"
Nov 02 17:57:54.902961 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:54.902831    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:57:59.069559 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.069526    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:57:59.158321 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158269    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.158531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158454    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.158531 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158505    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.158684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158586    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.158684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158625    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.158684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158670    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.158877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158733    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.159233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158894    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.159233 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.158941    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo\") pod \"76b1d399-8970-4edf-a253-6a032bc6e9c9\" (UID: \"76b1d399-8970-4edf-a253-6a032bc6e9c9\") "
Nov 02 17:57:59.160161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.159901    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:59.160161 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.160115    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.160416 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.160313    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:59.160517 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.160494    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:59.160911 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.160890    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:59.161045 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.160900    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:59.164709 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.164670    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:59.165151 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.165121    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "kube-api-access-g622r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:59.165435 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.165410    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:59.165515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.165431    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token" pod "76b1d399-8970-4edf-a253-6a032bc6e9c9" (UID: "76b1d399-8970-4edf-a253-6a032bc6e9c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:59.262027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.261993    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g622r\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-kube-api-access-g622r\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262020    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262030    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262039    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262051    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262059    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b1d399-8970-4edf-a253-6a032bc6e9c9-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262069    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.262463 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.262079    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b1d399-8970-4edf-a253-6a032bc6e9c9-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:57:59.579522 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.579487    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"]
Nov 02 17:57:59.580227 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.580212    5042 cpu_manager.go:414] "containerMap: removing container" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy"
Nov 02 17:57:59.580293 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.580229    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy"
Nov 02 17:57:59.580372 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.580363    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerName="istio-proxy"
Nov 02 17:57:59.585575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.585546    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.590335 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.590304    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-23a657\"/\"explicit-model-23a657-kserve-self-signed-certs\""
Nov 02 17:57:59.590931 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.590912    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-23a657\"/\"default-dockercfg-ccvp4\""
Nov 02 17:57:59.591676 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.591650    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-23a657\"/\"openshift-service-ca.crt\""
Nov 02 17:57:59.591807 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.591751    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-23a657\"/\"kube-root-ca.crt\""
Nov 02 17:57:59.596190 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.596170    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"]
Nov 02 17:57:59.664143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.664103    5042 generic.go:356] "Generic (PLEG): container finished" podID="76b1d399-8970-4edf-a253-6a032bc6e9c9" containerID="b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40" exitCode=0
Nov 02 17:57:59.664346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.664173    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"
Nov 02 17:57:59.664346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.664196    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" event={"ID":"76b1d399-8970-4edf-a253-6a032bc6e9c9","Type":"ContainerDied","Data":"b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40"}
Nov 02 17:57:59.664346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.664272    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6" event={"ID":"76b1d399-8970-4edf-a253-6a032bc6e9c9","Type":"ContainerDied","Data":"d3c396a71d21643dcaa45b9afc0227d4bab0a751d77f9fe6d36cac333ce21f88"}
Nov 02 17:57:59.664346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.664304    5042 scope.go:122] "RemoveContainer" containerID="b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40"
Nov 02 17:57:59.679016 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.678990    5042 scope.go:122] "RemoveContainer" containerID="b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40"
Nov 02 17:57:59.679429 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:57:59.679403    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40\": container with ID starting with b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40 not found: ID does not exist" containerID="b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40"
Nov 02 17:57:59.679527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.679441    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40"} err="failed to get container status \"b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40\": rpc error: code = NotFound desc = could not find container \"b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40\": container with ID starting with b9bfb536aeb06457b682b110570bef9ca4e3d6481031733cdd0623ff2f3d4f40 not found: ID does not exist"
Nov 02 17:57:59.691214 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.691165    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"]
Nov 02 17:57:59.697739 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.697709    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b7gw6"]
Nov 02 17:57:59.723024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.722990    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.723204 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.723054    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxzd\" (UniqueName: \"kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.723253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.723230    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.723350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.723335    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.723480 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.723462    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.723623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.723509    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.833336    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.833432    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxzd\" (UniqueName: \"kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.833537    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.833601    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.833743    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.833814    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.838240 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.834479    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.869419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.869375    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.894874 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.891859    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.952317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.946194    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.952317 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.950284    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:57:59.970919 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:57:59.958283    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxzd\" (UniqueName: \"kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd\") pod \"explicit-model-23a657-kserve-7b7c85fcc9-5s9qk\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") " pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:58:00.205427 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:00.205297    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:58:00.615766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:00.615257    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"]
Nov 02 17:58:00.622148 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:58:00.621975    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a48e162_6630_4c47_b38c_036ebd8ace51.slice/crio-de28901ae9d87c541e892888ec262b19b403dce9b2561c3048b71524a08c5161 WatchSource:0}: Error finding container de28901ae9d87c541e892888ec262b19b403dce9b2561c3048b71524a08c5161: Status 404 returned error can't find the container with id de28901ae9d87c541e892888ec262b19b403dce9b2561c3048b71524a08c5161
Nov 02 17:58:00.693100 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:00.693054    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk" event={"ID":"0a48e162-6630-4c47-b38c-036ebd8ace51","Type":"ContainerStarted","Data":"de28901ae9d87c541e892888ec262b19b403dce9b2561c3048b71524a08c5161"}
Nov 02 17:58:01.238137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.179901    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"]
Nov 02 17:58:01.312112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.273288    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.346881    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.346992    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347094    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347129    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347189    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347253    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347325    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347491    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.363886 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.347581    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hxjl\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.464618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.464457    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.516973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.516877    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hxjl\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.517256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.517235    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.517458 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.517440    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.517680 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.517662    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.517794 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.517780    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.517963 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.517948    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.518159 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.518144    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.518333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.518321    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.542349 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.542304    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.542529 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.478106    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"]
Nov 02 17:58:01.574231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.479134    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.574231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.552316    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.574231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.552942    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.574231 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.569685    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.934923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.917936    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.981055 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.972825    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.981401 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.975768    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.981571 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.976385    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hxjl\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl\") pod \"e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") " pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:01.987123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.986225    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b1d399-8970-4edf-a253-6a032bc6e9c9" path="/var/lib/kubelet/pods/76b1d399-8970-4edf-a253-6a032bc6e9c9/volumes"
Nov 02 17:58:01.987123 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:01.986761    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk" event={"ID":"0a48e162-6630-4c47-b38c-036ebd8ace51","Type":"ContainerStarted","Data":"263ef55a69c7c8760809c8f07ffb46f535f2299c050de3100be21e740ee72f30"}
Nov 02 17:58:02.069443 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.063491    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:58:02.159724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.089870    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:02.245865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.232247    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:02.269296 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.246514    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:58:02.329038 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.328992    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:02.353142 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.338413    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:02.353932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.353897    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:02.367866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.355906    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:02.440863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.421504    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk" podStartSLOduration=3.421470546 podStartE2EDuration="3.421470546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:02.062870321 +0000 UTC m=+1494.642279709" watchObservedRunningTime="2026-11-02 17:58:02.421470546 +0000 UTC m=+1495.000879933"
Nov 02 17:58:02.482490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.482445    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:02.542288 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.542075    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" podUID="d9e28883-4069-480e-b9c4-58529708f5da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": context deadline exceeded"
Nov 02 17:58:02.843126 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:02.843023    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:03.618900 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:58:03.618864    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d700c8_9e21_4d93_964f_aad473eb5dcd.slice/crio-45f195dde07fb0ffb54287cf4902f3248df272fbdbb556cbb5336ca04d3aa4e0 WatchSource:0}: Error finding container 45f195dde07fb0ffb54287cf4902f3248df272fbdbb556cbb5336ca04d3aa4e0: Status 404 returned error can't find the container with id 45f195dde07fb0ffb54287cf4902f3248df272fbdbb556cbb5336ca04d3aa4e0
Nov 02 17:58:03.648301 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:03.648257    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:03.648539 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:03.648520    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:03.648671 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:03.648658    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:03.665914 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:03.665298    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"]
Nov 02 17:58:03.905880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:03.905671    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:58:03.925223 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:03.925195    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:58:04.323675 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:04.323640    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" event={"ID":"d7d700c8-9e21-4d93-964f-aad473eb5dcd","Type":"ContainerStarted","Data":"a198bf285cc421c8ac3777debc03677a452dc1acdb504fffe4cdd461b85cc32d"}
Nov 02 17:58:04.324060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:04.324037    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" event={"ID":"d7d700c8-9e21-4d93-964f-aad473eb5dcd","Type":"ContainerStarted","Data":"45f195dde07fb0ffb54287cf4902f3248df272fbdbb556cbb5336ca04d3aa4e0"}
Nov 02 17:58:04.350644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:04.350583    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 17:58:04.443881 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:04.443083    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podStartSLOduration=3.44305633 podStartE2EDuration="3.44305633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:04.377376805 +0000 UTC m=+1496.956786232" watchObservedRunningTime="2026-11-02 17:58:04.44305633 +0000 UTC m=+1497.022465718"
Nov 02 17:58:05.247398 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:05.247364    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:05.249604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:05.249570    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 17:58:05.249758 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:05.249626    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 17:58:06.247578 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:06.247534    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 17:58:06.248108 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:06.247590    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 17:58:07.256428 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:07.256383    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 17:58:07.258722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:07.258681    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 17:58:07.754441 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:07.754406    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"]
Nov 02 17:58:08.160527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.160447    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"]
Nov 02 17:58:08.180868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.177967    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.262994 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.262955    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 17:58:08.263506 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.263018    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 17:58:08.264902 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.264830    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.265252 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.265173    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.265430 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.265402    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.265635 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.265618    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.265817 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.265799    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.266135 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.266114    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwd6\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.266466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.266442    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.266638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.266619    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.267057 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.267033    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.271657 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.271486    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"]
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369095    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369185    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369278    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwd6\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369363    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369390    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369407    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369460    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369514    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.369534    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.370323    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.371061    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.371600    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.371778    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.372864 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.372156    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.461867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.459199    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.466401 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.465927    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.466401 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.466264    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.468890 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.468862    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwd6\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:08.772205 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:08.767808    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:09.061926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.053932    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:58:09.061926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.058516    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 17:58:09.061926 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.061264    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:58:09.065933 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.064190    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 17:58:09.264977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.264158    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 17:58:09.264977 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.264210    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 17:58:09.640027 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.640001    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"]
Nov 02 17:58:09.645003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.644953    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:09.645157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.645079    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:09.645157 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:09.645125    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:10.247006 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.246951    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 02 17:58:10.247224 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.247038    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 02 17:58:10.665275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.665239    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" event={"ID":"048057a0-2665-4987-91ce-c40f6a5a1f24","Type":"ContainerStarted","Data":"f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e"}
Nov 02 17:58:10.665700 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.665311    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" event={"ID":"048057a0-2665-4987-91ce-c40f6a5a1f24","Type":"ContainerStarted","Data":"53e72060009589a40acbc910c7182f797c8bb92c46f13cc5b1e82351fdc8d56c"}
Nov 02 17:58:10.709450 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.705328    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podStartSLOduration=2.705306034 podStartE2EDuration="2.705306034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:10.700265459 +0000 UTC m=+1503.279674846" watchObservedRunningTime="2026-11-02 17:58:10.705306034 +0000 UTC m=+1503.284715421"
Nov 02 17:58:10.774901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.774868    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:10.776149 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.776123    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:10.776294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:10.776276    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:11.505865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.501530    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:11.505865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.503233    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:11.529870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.528782    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 17:58:11.597324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.596952    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:58:11.597324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.597023    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:11.794925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.781269    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:11.794925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:11.781336    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:12.809865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:12.809174    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:12.809865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:12.809334    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:13.779352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:13.779259    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:13.779352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:13.779320    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:14.791868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:14.777165    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:14.791868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:14.777256    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:15.795882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:15.794318    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:15.795882 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:15.794383    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:16.775466 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:16.775432    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 17:58:16.775698 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:16.775488    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 17:58:17.809934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.805346    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:17.809934 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.807373    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:17.815966 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.813190    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 17:58:17.864194 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.864161    5042 generic.go:356] "Generic (PLEG): container finished" podID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerID="2ac7909c9d0d111548febe3398dd73cc1d8522a97f0d2567c46e3452caa0dfab" exitCode=0
Nov 02 17:58:17.864610 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.864580    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" event={"ID":"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597","Type":"ContainerDied","Data":"2ac7909c9d0d111548febe3398dd73cc1d8522a97f0d2567c46e3452caa0dfab"}
Nov 02 17:58:17.865718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.865698    5042 scope.go:122] "RemoveContainer" containerID="2ac7909c9d0d111548febe3398dd73cc1d8522a97f0d2567c46e3452caa0dfab"
Nov 02 17:58:17.867725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.867692    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:17.867888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.867770    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:17.867888 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:17.867802    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:18.545221 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:18.545185    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:58:18.884998 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:18.884896    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" event={"ID":"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597","Type":"ContainerStarted","Data":"2987e5551f6a420321b0196d410c0c2cc0a04752f689760bc164972a51678759"}
Nov 02 17:58:19.338166 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:19.338136    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:58:19.338419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:19.338324    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:58:19.338523 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:19.338504    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 17:58:20.337403 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:20.337367    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:58:20.337936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:20.337428    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 17:58:21.337414 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:21.337372    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:58:21.337870 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:21.337427    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 17:58:22.349865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:22.344279    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:58:22.349865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:22.344346    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 17:58:22.844135 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:22.844093    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk" podUID="0a48e162-6630-4c47-b38c-036ebd8ace51" containerName="main" containerID="cri-o://263ef55a69c7c8760809c8f07ffb46f535f2299c050de3100be21e740ee72f30" gracePeriod=45
Nov 02 17:58:22.970924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:22.970759    5042 generic.go:356] "Generic (PLEG): container finished" podID="0a48e162-6630-4c47-b38c-036ebd8ace51" containerID="263ef55a69c7c8760809c8f07ffb46f535f2299c050de3100be21e740ee72f30" exitCode=0
Nov 02 17:58:22.970924 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:22.970804    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk" event={"ID":"0a48e162-6630-4c47-b38c-036ebd8ace51","Type":"ContainerDied","Data":"263ef55a69c7c8760809c8f07ffb46f535f2299c050de3100be21e740ee72f30"}
Nov 02 17:58:23.161734 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.161646    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"]
Nov 02 17:58:23.221865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.221177    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.223812 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.223769    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"]
Nov 02 17:58:23.338859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.337563    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:58:23.338859 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.337623    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341402    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341460    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341581    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341699    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmqf\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341820    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341883    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341916    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.341962    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.342350 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.342115    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.444087 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.444041    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.444990    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.444906    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445130 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445042    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445216 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445129    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445325 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445279    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445502    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445541    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.445593 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445578    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.446017 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.445645    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmqf\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.446563 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.446537    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.448220 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.448194    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.448516 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.448492    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.448744 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.448723    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.449884 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.449820    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.450028 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.450006    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.526952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.516433    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmqf\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.526952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.519302    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.547558 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.547532    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:58:23.561754 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.561720    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:23.649412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.648900    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir\") pod \"0a48e162-6630-4c47-b38c-036ebd8ace51\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") "
Nov 02 17:58:23.649412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.648957    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm\") pod \"0a48e162-6630-4c47-b38c-036ebd8ace51\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") "
Nov 02 17:58:23.649412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649106    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home\") pod \"0a48e162-6630-4c47-b38c-036ebd8ace51\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") "
Nov 02 17:58:23.649412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649255    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir" pod "0a48e162-6630-4c47-b38c-036ebd8ace51" (UID: "0a48e162-6630-4c47-b38c-036ebd8ace51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.649412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649266    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs\") pod \"0a48e162-6630-4c47-b38c-036ebd8ace51\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") "
Nov 02 17:58:23.649412 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649394    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache\") pod \"0a48e162-6630-4c47-b38c-036ebd8ace51\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") "
Nov 02 17:58:23.649896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649472    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd\" (UniqueName: \"kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd\") pod \"0a48e162-6630-4c47-b38c-036ebd8ace51\" (UID: \"0a48e162-6630-4c47-b38c-036ebd8ace51\") "
Nov 02 17:58:23.649896 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home" pod "0a48e162-6630-4c47-b38c-036ebd8ace51" (UID: "0a48e162-6630-4c47-b38c-036ebd8ace51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.650003 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.649918    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache" pod "0a48e162-6630-4c47-b38c-036ebd8ace51" (UID: "0a48e162-6630-4c47-b38c-036ebd8ace51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.650724 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.650699    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.650955 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.650938    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.651066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.651039    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.652515 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.652491    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd" pod "0a48e162-6630-4c47-b38c-036ebd8ace51" (UID: "0a48e162-6630-4c47-b38c-036ebd8ace51"). InnerVolumeSpecName "kube-api-access-2nxzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:23.653682 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.653650    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs" pod "0a48e162-6630-4c47-b38c-036ebd8ace51" (UID: "0a48e162-6630-4c47-b38c-036ebd8ace51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:58:23.653958 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.653923    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm" pod "0a48e162-6630-4c47-b38c-036ebd8ace51" (UID: "0a48e162-6630-4c47-b38c-036ebd8ace51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:23.755024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.753585    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxzd\" (UniqueName: \"kubernetes.io/projected/0a48e162-6630-4c47-b38c-036ebd8ace51-kube-api-access-2nxzd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.755024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.753614    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a48e162-6630-4c47-b38c-036ebd8ace51-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:23.755024 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:23.753629    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a48e162-6630-4c47-b38c-036ebd8ace51-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:24.012055 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:58:24.012020    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b66dbd_84d1_470b_b0f8_2e51fd7a2ca7.slice/crio-4e25d8901c40386efd2d917d50ad61185145e42302f8faf038a15d8112546197 WatchSource:0}: Error finding container 4e25d8901c40386efd2d917d50ad61185145e42302f8faf038a15d8112546197: Status 404 returned error can't find the container with id 4e25d8901c40386efd2d917d50ad61185145e42302f8faf038a15d8112546197
Nov 02 17:58:24.017063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.016625    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:24.017063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.016714    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:24.017063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.016761    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:24.022786 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.022755    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"]
Nov 02 17:58:24.026546 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.026513    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk" event={"ID":"0a48e162-6630-4c47-b38c-036ebd8ace51","Type":"ContainerDied","Data":"de28901ae9d87c541e892888ec262b19b403dce9b2561c3048b71524a08c5161"}
Nov 02 17:58:24.026684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.026583    5042 scope.go:122] "RemoveContainer" containerID="263ef55a69c7c8760809c8f07ffb46f535f2299c050de3100be21e740ee72f30"
Nov 02 17:58:24.026684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.026606    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"
Nov 02 17:58:24.077759 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.077637    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"]
Nov 02 17:58:24.088833 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.088790    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-23a657/explicit-model-23a657-kserve-7b7c85fcc9-5s9qk"]
Nov 02 17:58:24.339336 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.339241    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:58:24.339336 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:24.339306    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 02 17:58:25.028802 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.028764    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"]
Nov 02 17:58:25.036064 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.036031    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-407aa3fa-8gwsc"]
Nov 02 17:58:25.039636 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.039601    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" event={"ID":"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7","Type":"ContainerStarted","Data":"bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4"}
Nov 02 17:58:25.039818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.039669    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" event={"ID":"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7","Type":"ContainerStarted","Data":"4e25d8901c40386efd2d917d50ad61185145e42302f8faf038a15d8112546197"}
Nov 02 17:58:25.064256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.064204    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podStartSLOduration=2.064187594 podStartE2EDuration="2.064187594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:25.060600526 +0000 UTC m=+1517.640009912" watchObservedRunningTime="2026-11-02 17:58:25.064187594 +0000 UTC m=+1517.643596970"
Nov 02 17:58:25.347254 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.347168    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:58:25.347644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.347617    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:58:25.348575 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.348552    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:58:25.557791 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.557757    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a48e162-6630-4c47-b38c-036ebd8ace51" path="/var/lib/kubelet/pods/0a48e162-6630-4c47-b38c-036ebd8ace51/volumes"
Nov 02 17:58:25.558393 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.558377    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfc351c-8eeb-4cb4-8ae5-88babdc17a22" path="/var/lib/kubelet/pods/1cfc351c-8eeb-4cb4-8ae5-88babdc17a22/volumes"
Nov 02 17:58:25.562423 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.562394    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:25.564655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.564628    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 17:58:25.564760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:25.564681    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 17:58:26.562866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:26.562813    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 17:58:26.563301 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:26.562890    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 17:58:27.565338 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:27.565298    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 17:58:27.565968 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:27.565935    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 17:58:28.562508 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:28.562477    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 17:58:28.562684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:28.562526    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 17:58:29.562534 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:29.562499    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 17:58:29.563009 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:29.562569    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 17:58:30.564339 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:30.564301    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 02 17:58:30.564978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:30.564378    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 02 17:58:30.572184 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:30.572142    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"]
Nov 02 17:58:30.622654 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:30.622603    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"]
Nov 02 17:58:30.727978 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:30.727947    5042 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-explicit-23a657: configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:30.738126 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:30.737172    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume podName:feef2c75-54bb-4ac1-868e-b17a58cf5275 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:31.23714805 +0000 UTC m=+1523.816557428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume") pod "payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" (UID: "feef2c75-54bb-4ac1-868e-b17a58cf5275") : configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:31.259988 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:31.259915    5042 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-explicit-23a657: configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:31.260328 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:31.260306    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume podName:feef2c75-54bb-4ac1-868e-b17a58cf5275 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:32.260282981 +0000 UTC m=+1524.839692350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume") pod "payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" (UID: "feef2c75-54bb-4ac1-868e-b17a58cf5275") : configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:31.567829 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:31.567745    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:31.568392 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:31.568195    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:31.568950 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:31.568930    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 17:58:32.284966 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:32.284919    5042 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-explicit-23a657: configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:32.285190 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:32.285003    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume podName:feef2c75-54bb-4ac1-868e-b17a58cf5275 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:34.284986302 +0000 UTC m=+1526.864395676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume") pod "payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" (UID: "feef2c75-54bb-4ac1-868e-b17a58cf5275") : configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:33.030978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:33.030936    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"]
Nov 02 17:58:33.036208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:33.036176    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19d70f47-kknrm"]
Nov 02 17:58:33.548725 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:33.548690    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a40814f-638c-4aa5-9ecb-84dbf1a68acf" path="/var/lib/kubelet/pods/2a40814f-638c-4aa5-9ecb-84dbf1a68acf/volumes"
Nov 02 17:58:34.325647 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:34.325617    5042 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-explicit-23a657: configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:34.326096 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:34.325687    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume podName:feef2c75-54bb-4ac1-868e-b17a58cf5275 nodeName:}" failed. No retries permitted until 2026-11-02 17:58:38.325672686 +0000 UTC m=+1530.905082051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume") pod "payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" (UID: "feef2c75-54bb-4ac1-868e-b17a58cf5275") : configmap "payload-processing-plugins-e2e-explicit-23a657" not found
Nov 02 17:58:35.572783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:35.572731    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8" podUID="f2fbdf07-890f-45a5-8764-c7f4b901b1f6" containerName="payload-processing" containerID="cri-o://ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c" gracePeriod=25
Nov 02 17:58:35.960536 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:35.960507    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:58:36.062718 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.062683    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6\" (UniqueName: \"kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6\") pod \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") "
Nov 02 17:58:36.062943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.062750    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume\") pod \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\" (UID: \"f2fbdf07-890f-45a5-8764-c7f4b901b1f6\") "
Nov 02 17:58:36.063208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.063181    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume" pod "f2fbdf07-890f-45a5-8764-c7f4b901b1f6" (UID: "f2fbdf07-890f-45a5-8764-c7f4b901b1f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:36.064179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.064149    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:36.065424 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.065382    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6" pod "f2fbdf07-890f-45a5-8764-c7f4b901b1f6" (UID: "f2fbdf07-890f-45a5-8764-c7f4b901b1f6"). InnerVolumeSpecName "kube-api-access-8vrl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:36.165579 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.165540    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrl6\" (UniqueName: \"kubernetes.io/projected/f2fbdf07-890f-45a5-8764-c7f4b901b1f6-kube-api-access-8vrl6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:36.197752 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.197704    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" podUID="feef2c75-54bb-4ac1-868e-b17a58cf5275" containerName="payload-pre-processing" containerID="cri-o://8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626" gracePeriod=25
Nov 02 17:58:36.250111 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.250077    5042 generic.go:356] "Generic (PLEG): container finished" podID="f2fbdf07-890f-45a5-8764-c7f4b901b1f6" containerID="ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c" exitCode=0
Nov 02 17:58:36.250290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.250123    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"
Nov 02 17:58:36.250290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.250160    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8" event={"ID":"f2fbdf07-890f-45a5-8764-c7f4b901b1f6","Type":"ContainerDied","Data":"ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c"}
Nov 02 17:58:36.250290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.250222    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8" event={"ID":"f2fbdf07-890f-45a5-8764-c7f4b901b1f6","Type":"ContainerDied","Data":"585800ec1b775264cecf3321b4c2692f53c843ff2dc49cb44abf522572f2caec"}
Nov 02 17:58:36.250290 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.250245    5042 scope.go:122] "RemoveContainer" containerID="ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c"
Nov 02 17:58:36.303779 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.303736    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"]
Nov 02 17:58:36.310832 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.310796    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-23a657-59f8bdd477-7lhz8"]
Nov 02 17:58:36.316093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.316045    5042 scope.go:122] "RemoveContainer" containerID="ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c"
Nov 02 17:58:36.316514 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:36.316485    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c\": container with ID starting with ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c not found: ID does not exist" containerID="ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c"
Nov 02 17:58:36.316630 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.316527    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c"} err="failed to get container status \"ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c\": rpc error: code = NotFound desc = could not find container \"ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c\": container with ID starting with ef32f8b39130c28e9ac33ab4d8278a6ace9e840d50194d8c0ca2d4fd35a5440c not found: ID does not exist"
Nov 02 17:58:36.492623 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.492595    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:58:36.617867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.579649    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp\" (UniqueName: \"kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp\") pod \"feef2c75-54bb-4ac1-868e-b17a58cf5275\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") "
Nov 02 17:58:36.617867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.579911    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume\") pod \"feef2c75-54bb-4ac1-868e-b17a58cf5275\" (UID: \"feef2c75-54bb-4ac1-868e-b17a58cf5275\") "
Nov 02 17:58:36.617867 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.596640    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp" pod "feef2c75-54bb-4ac1-868e-b17a58cf5275" (UID: "feef2c75-54bb-4ac1-868e-b17a58cf5275"). InnerVolumeSpecName "kube-api-access-j9zfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:36.632557 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.632423    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume" pod "feef2c75-54bb-4ac1-868e-b17a58cf5275" (UID: "feef2c75-54bb-4ac1-868e-b17a58cf5275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:36.683603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.683566    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9zfp\" (UniqueName: \"kubernetes.io/projected/feef2c75-54bb-4ac1-868e-b17a58cf5275-kube-api-access-j9zfp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:36.683603 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:36.683601    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feef2c75-54bb-4ac1-868e-b17a58cf5275-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:37.322920 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.322880    5042 generic.go:356] "Generic (PLEG): container finished" podID="feef2c75-54bb-4ac1-868e-b17a58cf5275" containerID="8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626" exitCode=0
Nov 02 17:58:37.323767 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.323423    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"
Nov 02 17:58:37.337959 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.323458    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" event={"ID":"feef2c75-54bb-4ac1-868e-b17a58cf5275","Type":"ContainerDied","Data":"8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626"}
Nov 02 17:58:37.338170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.338025    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m" event={"ID":"feef2c75-54bb-4ac1-868e-b17a58cf5275","Type":"ContainerDied","Data":"ab43c47dfb17c0a3c87ae6f52ecaa09faf3809f0dcb28004c3c0b74b0cee07f5"}
Nov 02 17:58:37.338170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.338072    5042 scope.go:122] "RemoveContainer" containerID="8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626"
Nov 02 17:58:37.515865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.509000    5042 scope.go:122] "RemoveContainer" containerID="8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626"
Nov 02 17:58:37.515865 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:37.509897    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626\": container with ID starting with 8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626 not found: ID does not exist" containerID="8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626"
Nov 02 17:58:37.515865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.509935    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626"} err="failed to get container status \"8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626\": rpc error: code = NotFound desc = could not find container \"8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626\": container with ID starting with 8f3f28c83a8896789e7d4542bbefde9660e810013c524a1084ecba6565f0d626 not found: ID does not exist"
Nov 02 17:58:37.665208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.663395    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fbdf07-890f-45a5-8764-c7f4b901b1f6" path="/var/lib/kubelet/pods/f2fbdf07-890f-45a5-8764-c7f4b901b1f6/volumes"
Nov 02 17:58:37.665208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.664048    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"]
Nov 02 17:58:37.665208 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:37.664616    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-23a657-8564c67867-lkh9m"]
Nov 02 17:58:38.027334 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:38.027300    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"]
Nov 02 17:58:38.032436 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:38.032407    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67fa0b0-bh6jg"]
Nov 02 17:58:38.721595 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:38.721564    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"]
Nov 02 17:58:38.722140 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:38.721823    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" containerID="cri-o://2987e5551f6a420321b0196d410c0c2cc0a04752f689760bc164972a51678759" gracePeriod=30
Nov 02 17:58:39.545461 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.545424    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46c239f-2c0d-40d7-a5ff-25def0b5b3fd" path="/var/lib/kubelet/pods/b46c239f-2c0d-40d7-a5ff-25def0b5b3fd/volumes"
Nov 02 17:58:39.546025 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.546002    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feef2c75-54bb-4ac1-868e-b17a58cf5275" path="/var/lib/kubelet/pods/feef2c75-54bb-4ac1-868e-b17a58cf5275/volumes"
Nov 02 17:58:39.821275 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.821183    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"]
Nov 02 17:58:39.823319 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.823293    5042 cpu_manager.go:414] "containerMap: removing container" podUID="feef2c75-54bb-4ac1-868e-b17a58cf5275" containerName="payload-pre-processing"
Nov 02 17:58:39.823476 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.823463    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feef2c75-54bb-4ac1-868e-b17a58cf5275" containerName="payload-pre-processing"
Nov 02 17:58:39.823590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.823580    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f2fbdf07-890f-45a5-8764-c7f4b901b1f6" containerName="payload-processing"
Nov 02 17:58:39.823668 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.823658    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fbdf07-890f-45a5-8764-c7f4b901b1f6" containerName="payload-processing"
Nov 02 17:58:39.823784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.823774    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0a48e162-6630-4c47-b38c-036ebd8ace51" containerName="main"
Nov 02 17:58:39.823880 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.823870    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a48e162-6630-4c47-b38c-036ebd8ace51" containerName="main"
Nov 02 17:58:39.824156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.824144    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="feef2c75-54bb-4ac1-868e-b17a58cf5275" containerName="payload-pre-processing"
Nov 02 17:58:39.824253 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.824244    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fbdf07-890f-45a5-8764-c7f4b901b1f6" containerName="payload-processing"
Nov 02 17:58:39.824352 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.824321    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a48e162-6630-4c47-b38c-036ebd8ace51" containerName="main"
Nov 02 17:58:39.838661 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.835963    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.840499 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.840469    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"]
Nov 02 17:58:39.849855 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.849807    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-66d558-openshift-default-dockercfg-c7jss\""
Nov 02 17:58:39.954015 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.953970    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnp9\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954045    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954124    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954199 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954191    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954416    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954496    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954567 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954553    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954632 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954618    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:39.954801 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:39.954787    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.074617 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.074517    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.074932 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.074909    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.075118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.075101    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.075279 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.075264    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.075526 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.075509    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.075702 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.075688    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnp9\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.075818 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.075803    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.076012 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.075992    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.076162 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.076149    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.079314 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.079284    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.080129 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.080105    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.096916 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.096872    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.097642 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.097612    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.097824 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.097734    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.145868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.131363    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.145868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.140858    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnp9\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.145868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.141381    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.145868 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.142083    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo\") pod \"e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") " pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.183122 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.183071    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:40.375871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.371862    5042 patch_prober.go:29] interesting pod/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:58:40.375871 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.371933    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:58:40.665313 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.665278    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"]
Nov 02 17:58:40.665576 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:58:40.665545    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd4d7e7_7143_490d_93cc_e154e9360f12.slice/crio-93f793ab110f64d557628a95c72a1e7de2b6a847e28a055c6c289bbf854a9026 WatchSource:0}: Error finding container 93f793ab110f64d557628a95c72a1e7de2b6a847e28a055c6c289bbf854a9026: Status 404 returned error can't find the container with id 93f793ab110f64d557628a95c72a1e7de2b6a847e28a055c6c289bbf854a9026
Nov 02 17:58:40.669026 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.668991    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:40.669152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.669077    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:40.669152 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:40.669125    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:41.403114 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:41.402590    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" event={"ID":"4dd4d7e7-7143-490d-93cc-e154e9360f12","Type":"ContainerStarted","Data":"9628afa872c7d42ca3c947580a5489365fe2859095c31077b8b956ed7483b833"}
Nov 02 17:58:41.403114 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:41.402653    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" event={"ID":"4dd4d7e7-7143-490d-93cc-e154e9360f12","Type":"ContainerStarted","Data":"93f793ab110f64d557628a95c72a1e7de2b6a847e28a055c6c289bbf854a9026"}
Nov 02 17:58:41.437711 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:41.437654    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" podStartSLOduration=2.43763884 podStartE2EDuration="2.43763884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:41.433448412 +0000 UTC m=+1534.012857798" watchObservedRunningTime="2026-11-02 17:58:41.43763884 +0000 UTC m=+1534.017048225"
Nov 02 17:58:42.187543 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:42.186263    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:42.194544 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:42.194515    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:42.414510 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:42.414470    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:42.415722 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:42.415696    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:58:44.474618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.473029    5042 generic.go:356] "Generic (PLEG): container finished" podID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerID="2987e5551f6a420321b0196d410c0c2cc0a04752f689760bc164972a51678759" exitCode=0
Nov 02 17:58:44.474618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.474380    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" event={"ID":"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597","Type":"ContainerDied","Data":"2987e5551f6a420321b0196d410c0c2cc0a04752f689760bc164972a51678759"}
Nov 02 17:58:44.474618 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.474485    5042 scope.go:122] "RemoveContainer" containerID="2ac7909c9d0d111548febe3398dd73cc1d8522a97f0d2567c46e3452caa0dfab"
Nov 02 17:58:44.538750 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.538719    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:58:44.638703 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.638666    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.638889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.638815    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.638986 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.638965    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.639052 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639036    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.639172 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639128    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.639229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639173    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.639282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639267    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.639324 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639310    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.639371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639343    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc\") pod \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\" (UID: \"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597\") "
Nov 02 17:58:44.640125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.639643    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:44.640125 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.640043    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:44.640757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.640443    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.640757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.640469    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.640757 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.640638    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:44.641092 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.641045    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:44.641538 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.641511    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:44.644298 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.644278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:44.644431 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.644291    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:44.644714 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.644690    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:44.645063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.645038    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc" pod "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" (UID: "6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597"). InnerVolumeSpecName "kube-api-access-2lljc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742353    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742385    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742395    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742403    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742414    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742430    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lljc\" (UniqueName: \"kubernetes.io/projected/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-kube-api-access-2lljc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:44.742520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:44.742445    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:58:45.531555 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:45.531513    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs" event={"ID":"6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597","Type":"ContainerDied","Data":"eecc427a5f5fb72681e008cafaeee69cb8ad4fe66b737e153eaeb97dd9cd092b"}
Nov 02 17:58:45.532173 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:45.532154    5042 scope.go:122] "RemoveContainer" containerID="2987e5551f6a420321b0196d410c0c2cc0a04752f689760bc164972a51678759"
Nov 02 17:58:45.532468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:45.532448    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"
Nov 02 17:58:45.806005 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:45.805906    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"]
Nov 02 17:58:45.806256 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:45.806240    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-23a657-openshift-default-7db6744997-ctqqs"]
Nov 02 17:58:46.074925 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:46.070914    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"]
Nov 02 17:58:46.114093 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:46.114044    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7565068a-4tbm6"]
Nov 02 17:58:47.546062 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:47.546032    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6be90a-cf4c-4ec4-8190-03028b6f0fea" path="/var/lib/kubelet/pods/1f6be90a-cf4c-4ec4-8190-03028b6f0fea/volumes"
Nov 02 17:58:47.546581 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:47.546540    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" path="/var/lib/kubelet/pods/6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597/volumes"
Nov 02 17:58:48.545430 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:58:48.545376    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:58:51.046054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:51.044926    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"]
Nov 02 17:58:51.073568 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:51.073488    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfbf9bc2-hmwsv"]
Nov 02 17:58:51.546118 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:51.546086    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef542d8-9ca3-4414-85c2-6db6c1849193" path="/var/lib/kubelet/pods/eef542d8-9ca3-4414-85c2-6db6c1849193/volumes"
Nov 02 17:58:55.049873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:55.035659    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"]
Nov 02 17:58:55.049873 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:55.035976    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" podUID="4dd4d7e7-7143-490d-93cc-e154e9360f12" containerName="istio-proxy" containerID="cri-o://9628afa872c7d42ca3c947580a5489365fe2859095c31077b8b956ed7483b833" gracePeriod=30
Nov 02 17:58:56.422077 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:56.422018    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:58:56.422608 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:56.422110    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:58:57.417318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:57.417253    5042 patch_prober.go:29] interesting pod/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:58:57.417638 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:58:57.417594    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" podUID="4dd4d7e7-7143-490d-93cc-e154e9360f12" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:59:00.728315 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:00.728264    5042 generic.go:356] "Generic (PLEG): container finished" podID="4dd4d7e7-7143-490d-93cc-e154e9360f12" containerID="9628afa872c7d42ca3c947580a5489365fe2859095c31077b8b956ed7483b833" exitCode=0
Nov 02 17:59:00.728760 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:00.728341    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" event={"ID":"4dd4d7e7-7143-490d-93cc-e154e9360f12","Type":"ContainerDied","Data":"9628afa872c7d42ca3c947580a5489365fe2859095c31077b8b956ed7483b833"}
Nov 02 17:59:01.038530 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.038483    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"]
Nov 02 17:59:01.047783 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.046716    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f3d6a1a-jpklq"]
Nov 02 17:59:01.269031 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.269005    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:59:01.386647 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.386606    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.386901 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.386727    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.386978 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.386919    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.387063 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.387041    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.387189 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.387172    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.387264 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.387204    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.387327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.387301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.387434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.387359    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.387434 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.387399    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert\") pod \"4dd4d7e7-7143-490d-93cc-e154e9360f12\" (UID: \"4dd4d7e7-7143-490d-93cc-e154e9360f12\") "
Nov 02 17:59:01.389046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.388332    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:59:01.389046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.388525    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:59:01.389046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.388621    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.389046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.388832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:59:01.389046 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.388877    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:59:01.389600 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.389390    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:59:01.393294 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.393256    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:59:01.393449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.393389    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:59:01.393449 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.393393    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:59:01.393622 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.393591    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9" pod "4dd4d7e7-7143-490d-93cc-e154e9360f12" (UID: "4dd4d7e7-7143-490d-93cc-e154e9360f12"). InnerVolumeSpecName "kube-api-access-2lnp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:59:01.490520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490482    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490518    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lnp9\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-kube-api-access-2lnp9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490529    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490520 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490538    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490552    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dd4d7e7-7143-490d-93cc-e154e9360f12-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490563    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490571    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.490954 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.490582    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dd4d7e7-7143-490d-93cc-e154e9360f12-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 17:59:01.546354 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.546317    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b0d206-cb9b-4817-a081-26dd624e3f82" path="/var/lib/kubelet/pods/52b0d206-cb9b-4817-a081-26dd624e3f82/volumes"
Nov 02 17:59:01.739883 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.739808    5042 generic.go:356] "Generic (PLEG): container finished" podID="4650ea1b-88c4-4333-b553-27f88d815e90" containerID="c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab" exitCode=0
Nov 02 17:59:01.740345 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.739883    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" event={"ID":"4650ea1b-88c4-4333-b553-27f88d815e90","Type":"ContainerDied","Data":"c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"}
Nov 02 17:59:01.740427 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.740359    5042 scope.go:122] "RemoveContainer" containerID="c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"
Nov 02 17:59:01.741784 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.741760    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4" event={"ID":"4dd4d7e7-7143-490d-93cc-e154e9360f12","Type":"ContainerDied","Data":"93f793ab110f64d557628a95c72a1e7de2b6a847e28a055c6c289bbf854a9026"}
Nov 02 17:59:01.741943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.741818    5042 scope.go:122] "RemoveContainer" containerID="9628afa872c7d42ca3c947580a5489365fe2859095c31077b8b956ed7483b833"
Nov 02 17:59:01.741943 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.741774    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"
Nov 02 17:59:01.742346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.742320    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:01.742425 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.742405    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:01.742488 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.742460    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:01.818742 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.818711    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"]
Nov 02 17:59:01.828382 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:01.828346    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-66d558-openshift-default-f89bd98b5-6q5z4"]
Nov 02 17:59:02.048217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:02.048014    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"]
Nov 02 17:59:02.052747 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:02.052715    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c968bd4-jvhz4"]
Nov 02 17:59:02.819913 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:02.818302    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" event={"ID":"4650ea1b-88c4-4333-b553-27f88d815e90","Type":"ContainerStarted","Data":"49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f"}
Nov 02 17:59:03.437020 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:03.436984    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:59:03.437330 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:03.437305    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:03.437419 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:03.437364    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 17:59:03.546464 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:03.546432    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd4d7e7-7143-490d-93cc-e154e9360f12" path="/var/lib/kubelet/pods/4dd4d7e7-7143-490d-93cc-e154e9360f12/volumes"
Nov 02 17:59:03.547066 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:03.547050    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c76f28-8ec8-4347-8700-1294ac44f4b5" path="/var/lib/kubelet/pods/e8c76f28-8ec8-4347-8700-1294ac44f4b5/volumes"
Nov 02 17:59:04.452865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:04.450320    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:04.452865 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:04.450435    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 17:59:05.437229 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:05.437191    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:05.437456 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:05.437271    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 17:59:06.436289 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:06.436250    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:06.436814 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:06.436319    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 17:59:07.440284 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:07.440245    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:07.441258 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:07.441206    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 17:59:08.437051 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:08.437008    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:08.437263 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:08.437087    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Nov 02 17:59:09.442684 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:09.442649    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:59:09.443156 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:09.443104    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:59:09.444041 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:09.444019    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 17:59:10.639879 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:10.639829    5042 scope.go:122] "RemoveContainer" containerID="ae6cfc4c3e6ea667c26e38fd3338154d92b70fa20504ed9f7a44f60721d27f92"
Nov 02 17:59:13.026188 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:13.026149    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"]
Nov 02 17:59:13.030800 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:13.030773    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df6c2497-89h6x"]
Nov 02 17:59:13.547644 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:13.547606    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a22c7b-d696-4a22-9eb0-ca2d9e0d2659" path="/var/lib/kubelet/pods/34a22c7b-d696-4a22-9eb0-ca2d9e0d2659/volumes"
Nov 02 17:59:16.028295 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:16.028258    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"]
Nov 02 17:59:16.032993 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:16.032965    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09afa86-trgvk"]
Nov 02 17:59:17.546472 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:17.546435    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65dba72-19b0-41ce-b682-87c57cf2d0d3" path="/var/lib/kubelet/pods/d65dba72-19b0-41ce-b682-87c57cf2d0d3/volumes"
Nov 02 17:59:18.629521 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:59:18.629484    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:59:29.055604 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:29.055545    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"]
Nov 02 17:59:29.063519 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:29.063480    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-014ff898-nd5ft"]
Nov 02 17:59:29.549179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:29.549148    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e167f8b8-37c6-4fd3-be22-cb46c45ca6d5" path="/var/lib/kubelet/pods/e167f8b8-37c6-4fd3-be22-cb46c45ca6d5/volumes"
Nov 02 17:59:37.487866 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:37.486953    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:41.027333 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:41.027286    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"]
Nov 02 17:59:41.032468 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:41.032436    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7561eeb3-lkhpr"]
Nov 02 17:59:41.546257 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:41.546219    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac55011b-d072-49af-ae32-89dff378f8e8" path="/var/lib/kubelet/pods/ac55011b-d072-49af-ae32-89dff378f8e8/volumes"
Nov 02 17:59:45.502010 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.501975    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"]
Nov 02 17:59:45.503889 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.503863    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy"
Nov 02 17:59:45.504060 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504047    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy"
Nov 02 17:59:45.504143 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504134    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy"
Nov 02 17:59:45.504215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504205    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy"
Nov 02 17:59:45.504346 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504337    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4dd4d7e7-7143-490d-93cc-e154e9360f12" containerName="istio-proxy"
Nov 02 17:59:45.504417 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504407    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd4d7e7-7143-490d-93cc-e154e9360f12" containerName="istio-proxy"
Nov 02 17:59:45.504681 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504671    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy"
Nov 02 17:59:45.504774 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.504765    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd4d7e7-7143-490d-93cc-e154e9360f12" containerName="istio-proxy"
Nov 02 17:59:45.511112 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.511085    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.518448 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.518423    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0f0baf-openshift-default-dockercfg-pjqf9\""
Nov 02 17:59:45.522121 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.522093    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"]
Nov 02 17:59:45.542061 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.542025    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.757465    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.757677    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.757708    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.757782    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.757876    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.757967    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwm9\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.758089    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.764952 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.758144    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.887127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.887079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwm9\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.888572 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.888545    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.889217 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.889195    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.902895 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.902859    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.903179 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.903155    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.903505 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.903482    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.903663 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.903642    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.903872 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.903829    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.904068 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.904047    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.911170 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.911138    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.911670 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.911647    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.919877 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.912031    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.920512 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.889736    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:45.974387 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:45.921757    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:46.033230 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.033134    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:46.074260 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.074213    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:46.075235 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.075207    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwm9\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:46.075410 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.075325    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo\") pod \"e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") " pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:46.178999 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.178962    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:46.589042 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:59:46.589003    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fc31ec_cf6a_424d_8903_9a65607e3f27.slice/crio-b6437946179304cab6258a2b36972e78e111439fac7aa545761d581c925f4c81 WatchSource:0}: Error finding container b6437946179304cab6258a2b36972e78e111439fac7aa545761d581c925f4c81: Status 404 returned error can't find the container with id b6437946179304cab6258a2b36972e78e111439fac7aa545761d581c925f4c81
Nov 02 17:59:46.589455 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.589428    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"]
Nov 02 17:59:46.593116 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.593075    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:46.593206 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.593172    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:46.593247 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.593214    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:46.739878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.739799    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" event={"ID":"c5fc31ec-cf6a-424d-8903-9a65607e3f27","Type":"ContainerStarted","Data":"4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd"}
Nov 02 17:59:46.739878 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.739880    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" event={"ID":"c5fc31ec-cf6a-424d-8903-9a65607e3f27","Type":"ContainerStarted","Data":"b6437946179304cab6258a2b36972e78e111439fac7aa545761d581c925f4c81"}
Nov 02 17:59:46.764490 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:46.764414    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" podStartSLOduration=1.7643944390000001 podStartE2EDuration="1.764394439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:46.760896428 +0000 UTC m=+1599.340305818" watchObservedRunningTime="2026-11-02 17:59:46.764394439 +0000 UTC m=+1599.343803832"
Nov 02 17:59:47.180037 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:47.179994    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:47.181056 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:47.181024    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 17:59:47.181215 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:47.181095    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" podUID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 17:59:48.186043 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:48.186010    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:48.541640 ip-10-0-135-5 kubenswrapper[5042]: E1102 17:59:48.541602    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 17:59:48.768396 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:48.768363    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:48.769369 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:48.769344    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 17:59:50.424127 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.424086    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"]
Nov 02 17:59:50.425013 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.424995    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f72c7c2-6a6d-4c93-aedb-eb1a5fd19597" containerName="istio-proxy"
Nov 02 17:59:50.429150 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.429133    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:50.441643 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.441614    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-0f0baf-dockercfg-jhtr4\""
Nov 02 17:59:50.442000 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.441945    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-0f0baf\""
Nov 02 17:59:50.444134 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.444108    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"]
Nov 02 17:59:50.500863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.498043    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctkll\" (UniqueName: \"kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll\") pod \"payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:50.500863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.498298    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:50.723923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.601326    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"]
Nov 02 17:59:50.723923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.641498    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:50.723923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.641701    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctkll\" (UniqueName: \"kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll\") pod \"payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:50.723923 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.706461    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:50.924863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.924036    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"]
Nov 02 17:59:50.924863 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.924163    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:50.947137 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:50.947107    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctkll\" (UniqueName: \"kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll\") pod \"payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:51.049373 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.049283    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:51.070527 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.070476    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.071022 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.070988    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftrp\" (UniqueName: \"kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp\") pod \"payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.244697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.242312    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftrp\" (UniqueName: \"kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp\") pod \"payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.244697 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.242588    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.246973 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.243820    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.263962 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.263897    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftrp\" (UniqueName: \"kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp\") pod \"payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.360632 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:59:51.360598    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef9de6c_1ea5_456d_a03f_2c4d1c02b36a.slice/crio-1c90bc6a2e8444e60d3ff17cc3a8b5225ae4c72d5d438f0ec2e7642d15240ee6 WatchSource:0}: Error finding container 1c90bc6a2e8444e60d3ff17cc3a8b5225ae4c72d5d438f0ec2e7642d15240ee6: Status 404 returned error can't find the container with id 1c90bc6a2e8444e60d3ff17cc3a8b5225ae4c72d5d438f0ec2e7642d15240ee6
Nov 02 17:59:51.361054 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.361027    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"]
Nov 02 17:59:51.552318 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.552285    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.853028 ip-10-0-135-5 kubenswrapper[5042]: W1102 17:59:51.852995    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9536c99d_d79e_4177_b6a4_4d03ebab7a34.slice/crio-c2fa104b000a740a5d40ea40f01c7421900ce28ae27c7e4cafe5cc0e58e7b7ab WatchSource:0}: Error finding container c2fa104b000a740a5d40ea40f01c7421900ce28ae27c7e4cafe5cc0e58e7b7ab: Status 404 returned error can't find the container with id c2fa104b000a740a5d40ea40f01c7421900ce28ae27c7e4cafe5cc0e58e7b7ab
Nov 02 17:59:51.853655 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.853597    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"]
Nov 02 17:59:51.957945 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.957893    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm" event={"ID":"9536c99d-d79e-4177-b6a4-4d03ebab7a34","Type":"ContainerStarted","Data":"ce0dd4042bab32ca91c943a01408ae4ee0eb5772a1a5fda896053928798187e0"}
Nov 02 17:59:51.958153 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.957969    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm" event={"ID":"9536c99d-d79e-4177-b6a4-4d03ebab7a34","Type":"ContainerStarted","Data":"c2fa104b000a740a5d40ea40f01c7421900ce28ae27c7e4cafe5cc0e58e7b7ab"}
Nov 02 17:59:51.958282 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.958252    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 17:59:51.959819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.959794    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl" event={"ID":"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a","Type":"ContainerStarted","Data":"503e4f225c896af2d2cc8f16441e1f9baa1a17341057378a7833808d1a55c3c5"}
Nov 02 17:59:51.959819 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.959852    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl" event={"ID":"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a","Type":"ContainerStarted","Data":"1c90bc6a2e8444e60d3ff17cc3a8b5225ae4c72d5d438f0ec2e7642d15240ee6"}
Nov 02 17:59:51.960067 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.959894    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 17:59:51.985633 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:51.985573    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm" podStartSLOduration=1.985557076 podStartE2EDuration="1.985557076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:51.982084125 +0000 UTC m=+1604.561493530" watchObservedRunningTime="2026-11-02 17:59:51.985557076 +0000 UTC m=+1604.564966462"
Nov 02 17:59:52.010766 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:52.010699    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl" podStartSLOduration=2.010680925 podStartE2EDuration="2.010680925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:52.00774442 +0000 UTC m=+1604.587153827" watchObservedRunningTime="2026-11-02 17:59:52.010680925 +0000 UTC m=+1604.590090312"
Nov 02 17:59:57.032327 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:57.032287    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"]
Nov 02 17:59:57.037371 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:57.037334    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"]
Nov 02 17:59:57.042308 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:57.042277    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3993dad0-ghhht"]
Nov 02 17:59:57.047358 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:57.047330    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0434ab4-ds6bv"]
Nov 02 17:59:57.598936 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:57.595006    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a872d9c-5f74-4361-a14f-1914ca50de40" path="/var/lib/kubelet/pods/1a872d9c-5f74-4361-a14f-1914ca50de40/volumes"
Nov 02 17:59:57.600590 ip-10-0-135-5 kubenswrapper[5042]: I1102 17:59:57.600561    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b2f6bd-c213-409c-8885-517dd600d429" path="/var/lib/kubelet/pods/f3b2f6bd-c213-409c-8885-517dd600d429/volumes"
Nov 02 18:00:00.177003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.176964    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"]
Nov 02 18:00:00.183568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.183537    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"]
Nov 02 18:00:00.183764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.183646    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.189122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.189092    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"]
Nov 02 18:00:00.189311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.189181    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.193530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.193507    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:00:00.193772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.193756    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:00:00.200616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.200541    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"]
Nov 02 18:00:00.255939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.218556    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.255939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.247405    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:00:00.269365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.268595    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnz5z\" (UniqueName: \"kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.269365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.268690    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.269365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.268728    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2zc\" (UniqueName: \"kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.269365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.268954    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twdf\" (UniqueName: \"kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.269365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.268999    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.275070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.275037    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"]
Nov 02 18:00:00.277177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.277010    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.287111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.287084    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"]
Nov 02 18:00:00.287248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.287132    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"]
Nov 02 18:00:00.287248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.287223    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.291909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.291878    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.296909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.296887    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"]
Nov 02 18:00:00.325247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.325082    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"]
Nov 02 18:00:00.335783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.335663    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"]
Nov 02 18:00:00.343888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.343861    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"]
Nov 02 18:00:00.350050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.349877    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.356910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.355716    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"]
Nov 02 18:00:00.365220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.365192    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"]
Nov 02 18:00:00.365365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.365320    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.370884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.370855    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.376537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.376506    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"]
Nov 02 18:00:00.382100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.382075    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"]
Nov 02 18:00:00.387378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.387352    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"]
Nov 02 18:00:00.392061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.392022    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"]
Nov 02 18:00:00.393330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393297    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtvk4\" (UniqueName: \"kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.393462 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393437    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.393538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393515    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.393678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393654    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.393762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393697    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8l9\" (UniqueName: \"kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.393821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393770    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnz5z\" (UniqueName: \"kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.393895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393876    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.393949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393908    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2zc\" (UniqueName: \"kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.394011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.393992    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.394090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.394072    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xr48\" (UniqueName: \"kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.394664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.394641    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7td\" (UniqueName: \"kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.395485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.395462    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twdf\" (UniqueName: \"kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.395649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.395632    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.395827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.395807    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.396225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.395920    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.396747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.396717    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.397273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.397249    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.400542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.400521    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"]
Nov 02 18:00:00.400689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.400669    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.407893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.407073    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2zc\" (UniqueName: \"kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.408236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.408203    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnz5z\" (UniqueName: \"kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.409146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.409009    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.409919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.409897    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twdf\" (UniqueName: \"kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.411575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.411552    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"]
Nov 02 18:00:00.416540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.416516    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"]
Nov 02 18:00:00.497572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.497538    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzgt\" (UniqueName: \"kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.497791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.497614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.497791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.497745    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xr48\" (UniqueName: \"kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.497946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.497815    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7td\" (UniqueName: \"kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.497946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.497889    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.498155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498007    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.498155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498104    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtvk4\" (UniqueName: \"kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.498361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498333    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.498497 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498395    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.498497 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498407    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.498609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498551    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqhc\" (UniqueName: \"kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.498609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498602    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.498719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498641    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.498777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498718    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8l9\" (UniqueName: \"kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.498777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498730    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.498953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.498887    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skpnj\" (UniqueName: \"kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.499368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.499345    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.499491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.499345    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.512570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.512505    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtvk4\" (UniqueName: \"kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4\") pod \"maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.512570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.512505    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7td\" (UniqueName: \"kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td\") pod \"maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.513532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.513497    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xr48\" (UniqueName: \"kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.514245 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.514222    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8l9\" (UniqueName: \"kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.574373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.574330    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:00.582285 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.582252    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:00.605106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.604756    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skpnj\" (UniqueName: \"kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.605106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.604912    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzgt\" (UniqueName: \"kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.605260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.605173    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.606652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.606625    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.606928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.606683    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.606928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.606712    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqhc\" (UniqueName: \"kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.607593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.607560    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.607743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.607672    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.607828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.607790    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.611084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.611059    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:00.615315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.615195    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:00.622888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.622538    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzgt\" (UniqueName: \"kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:00.628869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.628778    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skpnj\" (UniqueName: \"kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.636761 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.636725    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:00.643766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.643669    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqhc\" (UniqueName: \"kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.669503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.669256    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:00.685635 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.685603    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:00.691569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.691495    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:00.753368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.753122    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:00.757640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:00.757599    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:01.119898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.119757    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"]
Nov 02 18:00:01.180445 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:01.179479    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d92a33_9ea9_47c7_8fb1_35804aad17af.slice/crio-603580e4dc83305d625b13899fc120b78bbe8bc304aa11c63af41e66b5011e77 WatchSource:0}: Error finding container 603580e4dc83305d625b13899fc120b78bbe8bc304aa11c63af41e66b5011e77: Status 404 returned error can't find the container with id 603580e4dc83305d625b13899fc120b78bbe8bc304aa11c63af41e66b5011e77
Nov 02 18:00:01.369260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.362231    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"]
Nov 02 18:00:01.626979 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:01.626949    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802a456f_0275_4373_9428_e5e62e98b942.slice/crio-762fb96faebfbad19d8beddc6cbcd4f8a27b2295ea6a79ec882081a559631e73 WatchSource:0}: Error finding container 762fb96faebfbad19d8beddc6cbcd4f8a27b2295ea6a79ec882081a559631e73: Status 404 returned error can't find the container with id 762fb96faebfbad19d8beddc6cbcd4f8a27b2295ea6a79ec882081a559631e73
Nov 02 18:00:01.643093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.642537    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"]
Nov 02 18:00:01.650906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.650868    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"]
Nov 02 18:00:01.787956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.787925    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"]
Nov 02 18:00:01.795944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.795913    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"]
Nov 02 18:00:01.834206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.834178    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"]
Nov 02 18:00:01.880571 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:01.880532    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9290f100_625e_45d0_b4ec_216838391fbd.slice/crio-b7b94cb3a4180b4ade03170af3e891f9b530b704260101678d8a53925c45a3de WatchSource:0}: Error finding container b7b94cb3a4180b4ade03170af3e891f9b530b704260101678d8a53925c45a3de: Status 404 returned error can't find the container with id b7b94cb3a4180b4ade03170af3e891f9b530b704260101678d8a53925c45a3de
Nov 02 18:00:01.890485 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:01.890450    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255db28e_64ea_4f77_83d3_6a8930bd00f3.slice/crio-bba5cfea375358dd450cc7aa61a86743e016c339fbbd418ddc15fed33db9fe14 WatchSource:0}: Error finding container bba5cfea375358dd450cc7aa61a86743e016c339fbbd418ddc15fed33db9fe14: Status 404 returned error can't find the container with id bba5cfea375358dd450cc7aa61a86743e016c339fbbd418ddc15fed33db9fe14
Nov 02 18:00:01.981212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:01.981171    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"]
Nov 02 18:00:02.040615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.039500    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"]
Nov 02 18:00:02.043205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.043030    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"]
Nov 02 18:00:02.050382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.050357    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"]
Nov 02 18:00:02.055613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.055586    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b70b4c74-qczsz"]
Nov 02 18:00:02.077316 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:02.076825    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e421df1_dfed_4d34_9ad6_6692ffa48115.slice/crio-633881ee0b030fb06001408f2e11ed2d742ed5d01d846b84e41e9792865f9eb2 WatchSource:0}: Error finding container 633881ee0b030fb06001408f2e11ed2d742ed5d01d846b84e41e9792865f9eb2: Status 404 returned error can't find the container with id 633881ee0b030fb06001408f2e11ed2d742ed5d01d846b84e41e9792865f9eb2
Nov 02 18:00:02.143762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.143655    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerStarted","Data":"73c628318d6905eec381cd8c8f57edf8ce95605e4dec0ff76a4265fd9711bc88"}
Nov 02 18:00:02.145529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.145477    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerStarted","Data":"bba5cfea375358dd450cc7aa61a86743e016c339fbbd418ddc15fed33db9fe14"}
Nov 02 18:00:02.147251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.147220    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerStarted","Data":"b7b94cb3a4180b4ade03170af3e891f9b530b704260101678d8a53925c45a3de"}
Nov 02 18:00:02.148570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.148533    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerStarted","Data":"762fb96faebfbad19d8beddc6cbcd4f8a27b2295ea6a79ec882081a559631e73"}
Nov 02 18:00:02.149770 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.149749    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerStarted","Data":"745ad27acc282cf6afb52e530789abf9d0dd3650de53be0876208c5f7cb4c455"}
Nov 02 18:00:02.151004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.150979    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerStarted","Data":"603580e4dc83305d625b13899fc120b78bbe8bc304aa11c63af41e66b5011e77"}
Nov 02 18:00:02.152238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.152213    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerStarted","Data":"274a4647ef4ecbe28fe9294c4a27c358dd2e05de278aaf2149557f135d7bfd30"}
Nov 02 18:00:02.153640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.153615    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerStarted","Data":"633881ee0b030fb06001408f2e11ed2d742ed5d01d846b84e41e9792865f9eb2"}
Nov 02 18:00:02.154990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.154971    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerStarted","Data":"0f849a2785dcd7d7c1963f82cd1416c50b8cafcafb2d546c872cdea809c01448"}
Nov 02 18:00:02.156151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.156131    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerStarted","Data":"05895fe1b5b900e63a0074e2d637c355422cfff5695990ee6d9e02826a1e1c09"}
Nov 02 18:00:02.972008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.971973    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 18:00:02.972974 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:02.972945    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 18:00:03.170576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.170541    5042 generic.go:356] "Generic (PLEG): container finished" podID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerID="717b7da7e758edca4fb54baf3203a05f521379c43fda55c1fa9d56fad282acc1" exitCode=60
Nov 02 18:00:03.170745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.170629    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerDied","Data":"717b7da7e758edca4fb54baf3203a05f521379c43fda55c1fa9d56fad282acc1"}
Nov 02 18:00:03.171005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.170986    5042 scope.go:122] "RemoveContainer" containerID="717b7da7e758edca4fb54baf3203a05f521379c43fda55c1fa9d56fad282acc1"
Nov 02 18:00:03.172852 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.172805    5042 generic.go:356] "Generic (PLEG): container finished" podID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerID="86d78b5b86fc3e7c74ced7aa313e0ecd111cc3bfca312f78d92dc2364702f967" exitCode=60
Nov 02 18:00:03.173581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.173387    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerDied","Data":"86d78b5b86fc3e7c74ced7aa313e0ecd111cc3bfca312f78d92dc2364702f967"}
Nov 02 18:00:03.173815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.173793    5042 scope.go:122] "RemoveContainer" containerID="86d78b5b86fc3e7c74ced7aa313e0ecd111cc3bfca312f78d92dc2364702f967"
Nov 02 18:00:03.179693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.179666    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerStarted","Data":"5861d595850050bdd0845efabf72bc2fa3c2de150ae13ffc3fd1bf3e0d7a93d9"}
Nov 02 18:00:03.182351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.182292    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerStarted","Data":"f30c32c0e5dd003018c8dec31f3db23b136d65976b6a3838a7f7939c739ecb30"}
Nov 02 18:00:03.189395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.189362    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerStarted","Data":"f99e4a6c29cee666f4f38c2b29bbc1608a1194de1fe695d3ddda1ed9d74ff333"}
Nov 02 18:00:03.195953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.195920    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerStarted","Data":"279df2f01a31a9bfb205609e1284170ce1519b2b7f58455c7b3ae136fc579970"}
Nov 02 18:00:03.197911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.197885    5042 generic.go:356] "Generic (PLEG): container finished" podID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerID="c609d474ee060e2854e5ecba1abf39cfe31d8a11b5ea4f400e6c287a37f65cb5" exitCode=60
Nov 02 18:00:03.198051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.197998    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerDied","Data":"c609d474ee060e2854e5ecba1abf39cfe31d8a11b5ea4f400e6c287a37f65cb5"}
Nov 02 18:00:03.198265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.198240    5042 scope.go:122] "RemoveContainer" containerID="c609d474ee060e2854e5ecba1abf39cfe31d8a11b5ea4f400e6c287a37f65cb5"
Nov 02 18:00:03.200612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.200585    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerStarted","Data":"d4190762f6eb0dd8e4f007622051c9b4fd76c1b45930c4507d9be68c3e14d1ae"}
Nov 02 18:00:03.202651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.202630    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerStarted","Data":"c049e1d70474122846a8b1b4fade5350dbff8b426ca9967ba33c7658b0579825"}
Nov 02 18:00:03.209854 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.209786    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" podStartSLOduration=3.209767784 podStartE2EDuration="3.209767784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.205194985 +0000 UTC m=+1615.784604372" watchObservedRunningTime="2026-11-02 18:00:03.209767784 +0000 UTC m=+1615.789177171"
Nov 02 18:00:03.216479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.216449    5042 generic.go:356] "Generic (PLEG): container finished" podID="802a456f-0275-4373-9428-e5e62e98b942" containerID="4848ad502817fc87d7a513b17b8d10718dd891ea6a491402b4fd9b3b9bc91d86" exitCode=60
Nov 02 18:00:03.216642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.216533    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerDied","Data":"4848ad502817fc87d7a513b17b8d10718dd891ea6a491402b4fd9b3b9bc91d86"}
Nov 02 18:00:03.217048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.217026    5042 scope.go:122] "RemoveContainer" containerID="4848ad502817fc87d7a513b17b8d10718dd891ea6a491402b4fd9b3b9bc91d86"
Nov 02 18:00:03.249868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.249800    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" podStartSLOduration=3.249785181 podStartE2EDuration="3.249785181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.248950944 +0000 UTC m=+1615.828360338" watchObservedRunningTime="2026-11-02 18:00:03.249785181 +0000 UTC m=+1615.829194583"
Nov 02 18:00:03.278243 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.277550    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" podStartSLOduration=3.277528313 podStartE2EDuration="3.277528313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.266990174 +0000 UTC m=+1615.846399561" watchObservedRunningTime="2026-11-02 18:00:03.277528313 +0000 UTC m=+1615.856937700"
Nov 02 18:00:03.362640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.362580    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" podStartSLOduration=3.3625640470000002 podStartE2EDuration="3.362564047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.323393103 +0000 UTC m=+1615.902802490" watchObservedRunningTime="2026-11-02 18:00:03.362564047 +0000 UTC m=+1615.941973430"
Nov 02 18:00:03.366350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.366299    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" podStartSLOduration=3.366285745 podStartE2EDuration="3.366285745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.348506277 +0000 UTC m=+1615.927915700" watchObservedRunningTime="2026-11-02 18:00:03.366285745 +0000 UTC m=+1615.945695178"
Nov 02 18:00:03.373778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.373695    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" podStartSLOduration=3.3736721210000002 podStartE2EDuration="3.373672121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:03.366172672 +0000 UTC m=+1615.945582069" watchObservedRunningTime="2026-11-02 18:00:03.373672121 +0000 UTC m=+1615.953081539"
Nov 02 18:00:03.549889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:03.549818    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713f720b-b35a-4079-8f66-0faaba6c5265" path="/var/lib/kubelet/pods/713f720b-b35a-4079-8f66-0faaba6c5265/volumes"
Nov 02 18:00:04.229748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.229710    5042 generic.go:356] "Generic (PLEG): container finished" podID="9290f100-625e-45d0-b4ec-216838391fbd" containerID="c049e1d70474122846a8b1b4fade5350dbff8b426ca9967ba33c7658b0579825" exitCode=60
Nov 02 18:00:04.230317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.229802    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerDied","Data":"c049e1d70474122846a8b1b4fade5350dbff8b426ca9967ba33c7658b0579825"}
Nov 02 18:00:04.230317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.230196    5042 scope.go:122] "RemoveContainer" containerID="c049e1d70474122846a8b1b4fade5350dbff8b426ca9967ba33c7658b0579825"
Nov 02 18:00:04.232371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.232344    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerStarted","Data":"3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8"}
Nov 02 18:00:04.235224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.235195    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerStarted","Data":"da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101"}
Nov 02 18:00:04.237353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.237324    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerStarted","Data":"7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330"}
Nov 02 18:00:04.239174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.239149    5042 generic.go:356] "Generic (PLEG): container finished" podID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerID="5861d595850050bdd0845efabf72bc2fa3c2de150ae13ffc3fd1bf3e0d7a93d9" exitCode=60
Nov 02 18:00:04.239281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.239217    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerDied","Data":"5861d595850050bdd0845efabf72bc2fa3c2de150ae13ffc3fd1bf3e0d7a93d9"}
Nov 02 18:00:04.239508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.239488    5042 scope.go:122] "RemoveContainer" containerID="5861d595850050bdd0845efabf72bc2fa3c2de150ae13ffc3fd1bf3e0d7a93d9"
Nov 02 18:00:04.240752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.240731    5042 generic.go:356] "Generic (PLEG): container finished" podID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerID="f30c32c0e5dd003018c8dec31f3db23b136d65976b6a3838a7f7939c739ecb30" exitCode=60
Nov 02 18:00:04.240966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.240807    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerDied","Data":"f30c32c0e5dd003018c8dec31f3db23b136d65976b6a3838a7f7939c739ecb30"}
Nov 02 18:00:04.241061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.241045    5042 scope.go:122] "RemoveContainer" containerID="f30c32c0e5dd003018c8dec31f3db23b136d65976b6a3838a7f7939c739ecb30"
Nov 02 18:00:04.242574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.242552    5042 generic.go:356] "Generic (PLEG): container finished" podID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerID="f99e4a6c29cee666f4f38c2b29bbc1608a1194de1fe695d3ddda1ed9d74ff333" exitCode=60
Nov 02 18:00:04.242659 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.242631    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerDied","Data":"f99e4a6c29cee666f4f38c2b29bbc1608a1194de1fe695d3ddda1ed9d74ff333"}
Nov 02 18:00:04.242912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.242896    5042 scope.go:122] "RemoveContainer" containerID="f99e4a6c29cee666f4f38c2b29bbc1608a1194de1fe695d3ddda1ed9d74ff333"
Nov 02 18:00:04.244502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.244482    5042 generic.go:356] "Generic (PLEG): container finished" podID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerID="279df2f01a31a9bfb205609e1284170ce1519b2b7f58455c7b3ae136fc579970" exitCode=60
Nov 02 18:00:04.245997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.244904    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerDied","Data":"279df2f01a31a9bfb205609e1284170ce1519b2b7f58455c7b3ae136fc579970"}
Nov 02 18:00:04.246093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.244950    5042 scope.go:122] "RemoveContainer" containerID="279df2f01a31a9bfb205609e1284170ce1519b2b7f58455c7b3ae136fc579970"
Nov 02 18:00:04.259069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.259029    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerStarted","Data":"d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059"}
Nov 02 18:00:04.261484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.261455    5042 generic.go:356] "Generic (PLEG): container finished" podID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerID="d4190762f6eb0dd8e4f007622051c9b4fd76c1b45930c4507d9be68c3e14d1ae" exitCode=60
Nov 02 18:00:04.261597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.261516    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerDied","Data":"d4190762f6eb0dd8e4f007622051c9b4fd76c1b45930c4507d9be68c3e14d1ae"}
Nov 02 18:00:04.261950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.261933    5042 scope.go:122] "RemoveContainer" containerID="d4190762f6eb0dd8e4f007622051c9b4fd76c1b45930c4507d9be68c3e14d1ae"
Nov 02 18:00:04.318978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.318918    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" podStartSLOduration=4.318897931 podStartE2EDuration="4.318897931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.312382869 +0000 UTC m=+1616.891792248" watchObservedRunningTime="2026-11-02 18:00:04.318897931 +0000 UTC m=+1616.898307320"
Nov 02 18:00:04.339472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.339411    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" podStartSLOduration=4.339390725 podStartE2EDuration="4.339390725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.335336303 +0000 UTC m=+1616.914745686" watchObservedRunningTime="2026-11-02 18:00:04.339390725 +0000 UTC m=+1616.918800111"
Nov 02 18:00:04.407672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.407610    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" podStartSLOduration=4.407588033 podStartE2EDuration="4.407588033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.401825078 +0000 UTC m=+1616.981234464" watchObservedRunningTime="2026-11-02 18:00:04.407588033 +0000 UTC m=+1616.986997420"
Nov 02 18:00:04.446821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:04.446762    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" podStartSLOduration=4.446747438 podStartE2EDuration="4.446747438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:04.440699813 +0000 UTC m=+1617.020109199" watchObservedRunningTime="2026-11-02 18:00:04.446747438 +0000 UTC m=+1617.026156823"
Nov 02 18:00:05.303863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.302097    5042 generic.go:356] "Generic (PLEG): container finished" podID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerID="d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059" exitCode=60
Nov 02 18:00:05.303863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.302165    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerDied","Data":"d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059"}
Nov 02 18:00:05.303863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.302242    5042 scope.go:122] "RemoveContainer" containerID="c609d474ee060e2854e5ecba1abf39cfe31d8a11b5ea4f400e6c287a37f65cb5"
Nov 02 18:00:05.303863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.302599    5042 scope.go:122] "RemoveContainer" containerID="d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059"
Nov 02 18:00:05.303863 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:05.303017    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc_odh-ai-gateway-infra(b7733536-09f3-4ed5-bbec-48eaea5e05e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4"
Nov 02 18:00:05.308391 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.308320    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerStarted","Data":"70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73"}
Nov 02 18:00:05.312381 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.311411    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerStarted","Data":"594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38"}
Nov 02 18:00:05.315026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.314815    5042 generic.go:356] "Generic (PLEG): container finished" podID="802a456f-0275-4373-9428-e5e62e98b942" containerID="3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8" exitCode=60
Nov 02 18:00:05.315026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.314963    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerDied","Data":"3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8"}
Nov 02 18:00:05.315592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.315292    5042 scope.go:122] "RemoveContainer" containerID="3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8"
Nov 02 18:00:05.315908 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:05.315715    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg_odh-ai-gateway-infra(802a456f-0275-4373-9428-e5e62e98b942)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" podUID="802a456f-0275-4373-9428-e5e62e98b942"
Nov 02 18:00:05.317957 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.317936    5042 generic.go:356] "Generic (PLEG): container finished" podID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerID="da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101" exitCode=60
Nov 02 18:00:05.318069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.318031    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerDied","Data":"da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101"}
Nov 02 18:00:05.318519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.318498    5042 scope.go:122] "RemoveContainer" containerID="da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101"
Nov 02 18:00:05.319050 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:05.319028    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww_odh-ai-gateway-infra(e009d895-ad21-4dc3-ba59-3337e595ed85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85"
Nov 02 18:00:05.330294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.330216    5042 generic.go:356] "Generic (PLEG): container finished" podID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerID="7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330" exitCode=60
Nov 02 18:00:05.330740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.330718    5042 scope.go:122] "RemoveContainer" containerID="7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330"
Nov 02 18:00:05.331247 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:05.331222    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt_odh-ai-gateway-infra(96d92a33-9ea9-47c7-8fb1-35804aad17af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af"
Nov 02 18:00:05.331503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.331482    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerDied","Data":"7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330"}
Nov 02 18:00:05.332428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.332409    5042 scope.go:122] "RemoveContainer" containerID="4848ad502817fc87d7a513b17b8d10718dd891ea6a491402b4fd9b3b9bc91d86"
Nov 02 18:00:05.341299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.341264    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerStarted","Data":"91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559"}
Nov 02 18:00:05.344477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.344449    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerStarted","Data":"d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c"}
Nov 02 18:00:05.347593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.347566    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerStarted","Data":"3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc"}
Nov 02 18:00:05.349244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.349218    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerStarted","Data":"b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3"}
Nov 02 18:00:05.354424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.354404    5042 scope.go:122] "RemoveContainer" containerID="717b7da7e758edca4fb54baf3203a05f521379c43fda55c1fa9d56fad282acc1"
Nov 02 18:00:05.370378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:05.370356    5042 scope.go:122] "RemoveContainer" containerID="86d78b5b86fc3e7c74ced7aa313e0ecd111cc3bfca312f78d92dc2364702f967"
Nov 02 18:00:06.361429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.361389    5042 scope.go:122] "RemoveContainer" containerID="3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8"
Nov 02 18:00:06.362001 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.361917    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg_odh-ai-gateway-infra(802a456f-0275-4373-9428-e5e62e98b942)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" podUID="802a456f-0275-4373-9428-e5e62e98b942"
Nov 02 18:00:06.363401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.363378    5042 scope.go:122] "RemoveContainer" containerID="da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101"
Nov 02 18:00:06.363779 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.363755    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww_odh-ai-gateway-infra(e009d895-ad21-4dc3-ba59-3337e595ed85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85"
Nov 02 18:00:06.367896 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.365792    5042 scope.go:122] "RemoveContainer" containerID="7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330"
Nov 02 18:00:06.367896 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.366543    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt_odh-ai-gateway-infra(96d92a33-9ea9-47c7-8fb1-35804aad17af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af"
Nov 02 18:00:06.369452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.368828    5042 generic.go:356] "Generic (PLEG): container finished" podID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerID="91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559" exitCode=60
Nov 02 18:00:06.369452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.368978    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerDied","Data":"91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559"}
Nov 02 18:00:06.369452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.369071    5042 scope.go:122] "RemoveContainer" containerID="5861d595850050bdd0845efabf72bc2fa3c2de150ae13ffc3fd1bf3e0d7a93d9"
Nov 02 18:00:06.369684 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.369565    5042 scope.go:122] "RemoveContainer" containerID="91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559"
Nov 02 18:00:06.370424 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.370240    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2_odh-ai-gateway-infra(c9f1b851-c125-41b3-81ae-3cbc1ec6e468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468"
Nov 02 18:00:06.382210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.382182    5042 generic.go:356] "Generic (PLEG): container finished" podID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerID="d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c" exitCode=60
Nov 02 18:00:06.382387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.382280    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerDied","Data":"d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c"}
Nov 02 18:00:06.382660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.382631    5042 scope.go:122] "RemoveContainer" containerID="d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c"
Nov 02 18:00:06.383100 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.383075    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz_odh-ai-gateway-infra(1e421df1-dfed-4d34-9ad6-6692ffa48115)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115"
Nov 02 18:00:06.384673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.384648    5042 generic.go:356] "Generic (PLEG): container finished" podID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerID="3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc" exitCode=60
Nov 02 18:00:06.384780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.384724    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerDied","Data":"3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc"}
Nov 02 18:00:06.385100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.385080    5042 scope.go:122] "RemoveContainer" containerID="3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc"
Nov 02 18:00:06.385478 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.385454    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26_odh-ai-gateway-infra(1f8c0fbc-055c-4516-bfe3-5efbdeb91a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06"
Nov 02 18:00:06.389492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.389464    5042 generic.go:356] "Generic (PLEG): container finished" podID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerID="b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3" exitCode=60
Nov 02 18:00:06.389631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.389554    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerDied","Data":"b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3"}
Nov 02 18:00:06.389930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.389911    5042 scope.go:122] "RemoveContainer" containerID="b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3"
Nov 02 18:00:06.390369 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.390343    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d_odh-ai-gateway-infra(bc955a92-f1af-4fd7-bbb4-c1a53691a7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2"
Nov 02 18:00:06.392543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.392519    5042 scope.go:122] "RemoveContainer" containerID="d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059"
Nov 02 18:00:06.392956 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.392931    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc_odh-ai-gateway-infra(b7733536-09f3-4ed5-bbec-48eaea5e05e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4"
Nov 02 18:00:06.395565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.395534    5042 generic.go:356] "Generic (PLEG): container finished" podID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerID="70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73" exitCode=60
Nov 02 18:00:06.395673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.395641    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerDied","Data":"70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73"}
Nov 02 18:00:06.395943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.395928    5042 scope.go:122] "RemoveContainer" containerID="70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73"
Nov 02 18:00:06.396308 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.396288    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp_odh-ai-gateway-infra(255db28e-64ea-4f77-83d3-6a8930bd00f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3"
Nov 02 18:00:06.397621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.397603    5042 scope.go:122] "RemoveContainer" containerID="f30c32c0e5dd003018c8dec31f3db23b136d65976b6a3838a7f7939c739ecb30"
Nov 02 18:00:06.397991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.397971    5042 generic.go:356] "Generic (PLEG): container finished" podID="9290f100-625e-45d0-b4ec-216838391fbd" containerID="594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38" exitCode=60
Nov 02 18:00:06.398077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.398038    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerDied","Data":"594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38"}
Nov 02 18:00:06.398365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.398335    5042 scope.go:122] "RemoveContainer" containerID="594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38"
Nov 02 18:00:06.398674 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:06.398656    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb_odh-ai-gateway-infra(9290f100-625e-45d0-b4ec-216838391fbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" podUID="9290f100-625e-45d0-b4ec-216838391fbd"
Nov 02 18:00:06.414593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.414430    5042 scope.go:122] "RemoveContainer" containerID="f99e4a6c29cee666f4f38c2b29bbc1608a1194de1fe695d3ddda1ed9d74ff333"
Nov 02 18:00:06.434025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.434000    5042 scope.go:122] "RemoveContainer" containerID="279df2f01a31a9bfb205609e1284170ce1519b2b7f58455c7b3ae136fc579970"
Nov 02 18:00:06.451390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.451365    5042 scope.go:122] "RemoveContainer" containerID="d4190762f6eb0dd8e4f007622051c9b4fd76c1b45930c4507d9be68c3e14d1ae"
Nov 02 18:00:06.467707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:06.467684    5042 scope.go:122] "RemoveContainer" containerID="c049e1d70474122846a8b1b4fade5350dbff8b426ca9967ba33c7658b0579825"
Nov 02 18:00:07.409908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:07.409823    5042 scope.go:122] "RemoveContainer" containerID="70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73"
Nov 02 18:00:07.410392 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:07.410332    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp_odh-ai-gateway-infra(255db28e-64ea-4f77-83d3-6a8930bd00f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3"
Nov 02 18:00:07.411768 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:07.411752    5042 scope.go:122] "RemoveContainer" containerID="594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38"
Nov 02 18:00:07.412145 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:07.412123    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb_odh-ai-gateway-infra(9290f100-625e-45d0-b4ec-216838391fbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" podUID="9290f100-625e-45d0-b4ec-216838391fbd"
Nov 02 18:00:07.413645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:07.413626    5042 scope.go:122] "RemoveContainer" containerID="91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559"
Nov 02 18:00:07.414005 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:07.413981    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2_odh-ai-gateway-infra(c9f1b851-c125-41b3-81ae-3cbc1ec6e468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468"
Nov 02 18:00:07.415455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:07.415436    5042 scope.go:122] "RemoveContainer" containerID="d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c"
Nov 02 18:00:07.415735 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:07.415712    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz_odh-ai-gateway-infra(1e421df1-dfed-4d34-9ad6-6692ffa48115)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115"
Nov 02 18:00:07.417355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:07.417338    5042 scope.go:122] "RemoveContainer" containerID="3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc"
Nov 02 18:00:07.417646 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:07.417626    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26_odh-ai-gateway-infra(1f8c0fbc-055c-4516-bfe3-5efbdeb91a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06"
Nov 02 18:00:07.419101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:07.419083    5042 scope.go:122] "RemoveContainer" containerID="b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3"
Nov 02 18:00:07.419346 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:07.419325    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d_odh-ai-gateway-infra(bc955a92-f1af-4fd7-bbb4-c1a53691a7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2"
Nov 02 18:00:08.032113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:08.032075    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"]
Nov 02 18:00:08.037325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:08.037298    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6bfaf27-49r5g"]
Nov 02 18:00:09.039900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:09.039819    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"]
Nov 02 18:00:09.045426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:09.045385    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9a3b56a-cnclg"]
Nov 02 18:00:09.604491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:09.604451    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6" path="/var/lib/kubelet/pods/5cbd3ead-5ba3-4f0b-afb7-8bb637979dc6/volumes"
Nov 02 18:00:09.605279 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:09.605256    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0948114-a0a8-406c-b596-eb74f7309623" path="/var/lib/kubelet/pods/d0948114-a0a8-406c-b596-eb74f7309623/volumes"
Nov 02 18:00:10.801999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.801957    5042 scope.go:122] "RemoveContainer" containerID="22b85617e831fe3c4648a8b6dd81bb7cc79d11c0207f54735c6bc10ac13b51ad"
Nov 02 18:00:10.814998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.814976    5042 scope.go:122] "RemoveContainer" containerID="7e9285352efab342ade7e8f729f4a233ae7d4fe3bc4a5ed09a8eddc84acaee46"
Nov 02 18:00:10.828492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.828462    5042 scope.go:122] "RemoveContainer" containerID="c5d6e3a9a42a16f9c9dba875fc48a9bb027678abc15d802b084905e95d982314"
Nov 02 18:00:10.844149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.844115    5042 scope.go:122] "RemoveContainer" containerID="9164b27b9fe16d7eb869989ada8a4ac509567c80c785cfa8a88dda1a718a9b9a"
Nov 02 18:00:10.858604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.858580    5042 scope.go:122] "RemoveContainer" containerID="af629ae66792290d9664bd748cf7e6afbb866c326e66de4a043772bb547da9ee"
Nov 02 18:00:10.872536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.872513    5042 scope.go:122] "RemoveContainer" containerID="880131df7a47652ab9335230be657f288ed619e6eff2523f79a5344eb5e44c2d"
Nov 02 18:00:10.885712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.885689    5042 scope.go:122] "RemoveContainer" containerID="7eba7512cccdb1d71280a2f4d7a8c5ca8e5889345b39c2bb56b7d6f82b1c9eba"
Nov 02 18:00:10.899278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:10.899249    5042 scope.go:122] "RemoveContainer" containerID="e5816443496710226223ae940cd1f39a9acbe6d8494a774776c9f882036b753e"
Nov 02 18:00:15.536045 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:15.536014    5042 scope.go:122] "RemoveContainer" containerID="da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101"
Nov 02 18:00:16.528136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.528101    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerStarted","Data":"c758ab08051c9fa3cdb258a96d8fdbbff9e9ad6ef2124923ec1907b658e624ff"}
Nov 02 18:00:16.536108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.536079    5042 scope.go:122] "RemoveContainer" containerID="d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059"
Nov 02 18:00:16.536662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.536638    5042 scope.go:122] "RemoveContainer" containerID="b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3"
Nov 02 18:00:16.536761 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.536731    5042 scope.go:122] "RemoveContainer" containerID="70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73"
Nov 02 18:00:16.537229 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.536882    5042 scope.go:122] "RemoveContainer" containerID="d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c"
Nov 02 18:00:16.537229 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.537172    5042 scope.go:122] "RemoveContainer" containerID="91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559"
Nov 02 18:00:16.537328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:16.537307    5042 scope.go:122] "RemoveContainer" containerID="594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38"
Nov 02 18:00:17.542404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.542377    5042 scope.go:122] "RemoveContainer" containerID="3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8"
Nov 02 18:00:17.542922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.542786    5042 scope.go:122] "RemoveContainer" containerID="3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc"
Nov 02 18:00:17.548274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.548236    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerStarted","Data":"740dbc1e4cf265e91727cf216364280d06d65a30bfd64e4678ea5b421d6d7dd6"}
Nov 02 18:00:17.548419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.548296    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerStarted","Data":"191c385a66a5a319143805188c5aa0f830655179e3b9728c8e8f751685728a7d"}
Nov 02 18:00:17.552685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.552654    5042 generic.go:356] "Generic (PLEG): container finished" podID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerID="c758ab08051c9fa3cdb258a96d8fdbbff9e9ad6ef2124923ec1907b658e624ff" exitCode=60
Nov 02 18:00:17.552822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.552732    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerDied","Data":"c758ab08051c9fa3cdb258a96d8fdbbff9e9ad6ef2124923ec1907b658e624ff"}
Nov 02 18:00:17.552822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.552795    5042 scope.go:122] "RemoveContainer" containerID="da4d6314b902deb724949807f079aa9b207dcba9028f7d60e453b53f90f84101"
Nov 02 18:00:17.553190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.553174    5042 scope.go:122] "RemoveContainer" containerID="c758ab08051c9fa3cdb258a96d8fdbbff9e9ad6ef2124923ec1907b658e624ff"
Nov 02 18:00:17.553492 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:17.553468    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww_odh-ai-gateway-infra(e009d895-ad21-4dc3-ba59-3337e595ed85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85"
Nov 02 18:00:17.555607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.555579    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerStarted","Data":"f4e99375c7a8be621ae9f7b539205564e11b9d7c4082299f100e4ce0ef8286fb"}
Nov 02 18:00:17.558557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.558527    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerStarted","Data":"b9946e629deda742604e6d43f6f1a933e674723f6080e0eb243ac13b4d02798a"}
Nov 02 18:00:17.561972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.561935    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerStarted","Data":"d802dfce44b243085ad7c9007064495726b97ade08684ef2ed267c65e77c0160"}
Nov 02 18:00:17.564636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.564610    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerStarted","Data":"cfd2906e5b80985625eb32bd9e843ef7d805c97a6584fe2f28e211e213a5d574"}
Nov 02 18:00:17.565474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:17.565448    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"]
Nov 02 18:00:18.536211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.536179    5042 scope.go:122] "RemoveContainer" containerID="7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330"
Nov 02 18:00:18.543269 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.543233    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:00:18.576608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.576572    5042 generic.go:356] "Generic (PLEG): container finished" podID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerID="cfd2906e5b80985625eb32bd9e843ef7d805c97a6584fe2f28e211e213a5d574" exitCode=60
Nov 02 18:00:18.576808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.576732    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerDied","Data":"cfd2906e5b80985625eb32bd9e843ef7d805c97a6584fe2f28e211e213a5d574"}
Nov 02 18:00:18.576808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.576791    5042 scope.go:122] "RemoveContainer" containerID="d9fb09270eec89bbeef1a9b5db6df91d4390ea04154e11ffa940b93a4986e059"
Nov 02 18:00:18.577364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.577346    5042 scope.go:122] "RemoveContainer" containerID="cfd2906e5b80985625eb32bd9e843ef7d805c97a6584fe2f28e211e213a5d574"
Nov 02 18:00:18.577833 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.577808    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc_odh-ai-gateway-infra(b7733536-09f3-4ed5-bbec-48eaea5e05e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4"
Nov 02 18:00:18.586122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.586066    5042 generic.go:356] "Generic (PLEG): container finished" podID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerID="740dbc1e4cf265e91727cf216364280d06d65a30bfd64e4678ea5b421d6d7dd6" exitCode=60
Nov 02 18:00:18.586424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.586242    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerDied","Data":"740dbc1e4cf265e91727cf216364280d06d65a30bfd64e4678ea5b421d6d7dd6"}
Nov 02 18:00:18.587008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.586967    5042 scope.go:122] "RemoveContainer" containerID="740dbc1e4cf265e91727cf216364280d06d65a30bfd64e4678ea5b421d6d7dd6"
Nov 02 18:00:18.587429 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.587401    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp_odh-ai-gateway-infra(255db28e-64ea-4f77-83d3-6a8930bd00f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3"
Nov 02 18:00:18.588142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.588120    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"]
Nov 02 18:00:18.589573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.589544    5042 generic.go:356] "Generic (PLEG): container finished" podID="9290f100-625e-45d0-b4ec-216838391fbd" containerID="191c385a66a5a319143805188c5aa0f830655179e3b9728c8e8f751685728a7d" exitCode=60
Nov 02 18:00:18.589692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.589664    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerDied","Data":"191c385a66a5a319143805188c5aa0f830655179e3b9728c8e8f751685728a7d"}
Nov 02 18:00:18.591428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.591409    5042 scope.go:122] "RemoveContainer" containerID="191c385a66a5a319143805188c5aa0f830655179e3b9728c8e8f751685728a7d"
Nov 02 18:00:18.592241 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.592208    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb_odh-ai-gateway-infra(9290f100-625e-45d0-b4ec-216838391fbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" podUID="9290f100-625e-45d0-b4ec-216838391fbd"
Nov 02 18:00:18.593664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.593637    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerStarted","Data":"025b493787e635fdcd7d3ddddc7362a36061341081458a2ba03eed86b32421d4"}
Nov 02 18:00:18.604598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.602426    5042 generic.go:356] "Generic (PLEG): container finished" podID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerID="f4e99375c7a8be621ae9f7b539205564e11b9d7c4082299f100e4ce0ef8286fb" exitCode=60
Nov 02 18:00:18.604598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.602505    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerDied","Data":"f4e99375c7a8be621ae9f7b539205564e11b9d7c4082299f100e4ce0ef8286fb"}
Nov 02 18:00:18.604598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.604110    5042 scope.go:122] "RemoveContainer" containerID="f4e99375c7a8be621ae9f7b539205564e11b9d7c4082299f100e4ce0ef8286fb"
Nov 02 18:00:18.604598 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.604545    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2_odh-ai-gateway-infra(c9f1b851-c125-41b3-81ae-3cbc1ec6e468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468"
Nov 02 18:00:18.605085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.604972    5042 scope.go:122] "RemoveContainer" containerID="70e03863e22a46fd6b8609450699acd949e9704da24ec0b168a0fa5adaa52c73"
Nov 02 18:00:18.608230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.607266    5042 generic.go:356] "Generic (PLEG): container finished" podID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerID="b9946e629deda742604e6d43f6f1a933e674723f6080e0eb243ac13b4d02798a" exitCode=60
Nov 02 18:00:18.608230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.607348    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerDied","Data":"b9946e629deda742604e6d43f6f1a933e674723f6080e0eb243ac13b4d02798a"}
Nov 02 18:00:18.608230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.607782    5042 scope.go:122] "RemoveContainer" containerID="b9946e629deda742604e6d43f6f1a933e674723f6080e0eb243ac13b4d02798a"
Nov 02 18:00:18.608230 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.608174    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz_odh-ai-gateway-infra(1e421df1-dfed-4d34-9ad6-6692ffa48115)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115"
Nov 02 18:00:18.613687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.613659    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerStarted","Data":"af07338de74b4bf65a704da6f99a854b1b49fd6138b7847c153698b192b72126"}
Nov 02 18:00:18.618531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.618483    5042 generic.go:356] "Generic (PLEG): container finished" podID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerID="d802dfce44b243085ad7c9007064495726b97ade08684ef2ed267c65e77c0160" exitCode=60
Nov 02 18:00:18.618678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.618535    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerDied","Data":"d802dfce44b243085ad7c9007064495726b97ade08684ef2ed267c65e77c0160"}
Nov 02 18:00:18.619075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.619058    5042 scope.go:122] "RemoveContainer" containerID="d802dfce44b243085ad7c9007064495726b97ade08684ef2ed267c65e77c0160"
Nov 02 18:00:18.619433 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:18.619415    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d_odh-ai-gateway-infra(bc955a92-f1af-4fd7-bbb4-c1a53691a7f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2"
Nov 02 18:00:18.624044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.623997    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"]
Nov 02 18:00:18.634713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.634681    5042 scope.go:122] "RemoveContainer" containerID="594fa89bba87beb64771e72094e0e14c517cc836be2b1277e540c38d45065d38"
Nov 02 18:00:18.636114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.636054    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"]
Nov 02 18:00:18.654585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.654557    5042 scope.go:122] "RemoveContainer" containerID="91b5c779e7e062f5210e8a1476055f30c581acac0b9ac9c23b0d1f17fed1b559"
Nov 02 18:00:18.659366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.659343    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"]
Nov 02 18:00:18.678927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.678781    5042 scope.go:122] "RemoveContainer" containerID="d3b8562a21dbfd27e119c56584ba77aac1fbcaba67317ffb6e15d1688770ec3c"
Nov 02 18:00:18.685617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.685213    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"]
Nov 02 18:00:18.718204 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.718174    5042 scope.go:122] "RemoveContainer" containerID="b45753c088254ded57536bb03ff3bff6905b83df25e5e8a1eaa98520211e16d3"
Nov 02 18:00:18.746066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.745980    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"]
Nov 02 18:00:18.870415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.870388    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:18.941892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.941859    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca\") pod \"e009d895-ad21-4dc3-ba59-3337e595ed85\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") "
Nov 02 18:00:18.942127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.941989    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc\" (UniqueName: \"kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc\") pod \"e009d895-ad21-4dc3-ba59-3337e595ed85\" (UID: \"e009d895-ad21-4dc3-ba59-3337e595ed85\") "
Nov 02 18:00:18.942280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.942254    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca" pod "e009d895-ad21-4dc3-ba59-3337e595ed85" (UID: "e009d895-ad21-4dc3-ba59-3337e595ed85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:18.942918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.942892    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e009d895-ad21-4dc3-ba59-3337e595ed85-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:18.944471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:18.944449    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc" pod "e009d895-ad21-4dc3-ba59-3337e595ed85" (UID: "e009d895-ad21-4dc3-ba59-3337e595ed85"). InnerVolumeSpecName "kube-api-access-hs2zc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:19.044519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.044419    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2zc\" (UniqueName: \"kubernetes.io/projected/e009d895-ad21-4dc3-ba59-3337e595ed85-kube-api-access-hs2zc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:19.641274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.641237    5042 generic.go:356] "Generic (PLEG): container finished" podID="802a456f-0275-4373-9428-e5e62e98b942" containerID="025b493787e635fdcd7d3ddddc7362a36061341081458a2ba03eed86b32421d4" exitCode=60
Nov 02 18:00:19.641689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.641345    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerDied","Data":"025b493787e635fdcd7d3ddddc7362a36061341081458a2ba03eed86b32421d4"}
Nov 02 18:00:19.641689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.641408    5042 scope.go:122] "RemoveContainer" containerID="3f89c9cdaff3fdb425598d8ce8fd73587cdac81217a0a25c80851bae3f6590b8"
Nov 02 18:00:19.641992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.641973    5042 scope.go:122] "RemoveContainer" containerID="025b493787e635fdcd7d3ddddc7362a36061341081458a2ba03eed86b32421d4"
Nov 02 18:00:19.642410 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:19.642386    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg_odh-ai-gateway-infra(802a456f-0275-4373-9428-e5e62e98b942)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" podUID="802a456f-0275-4373-9428-e5e62e98b942"
Nov 02 18:00:19.644578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.644555    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"
Nov 02 18:00:19.644704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.644596    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww" event={"ID":"e009d895-ad21-4dc3-ba59-3337e595ed85","Type":"ContainerDied","Data":"745ad27acc282cf6afb52e530789abf9d0dd3650de53be0876208c5f7cb4c455"}
Nov 02 18:00:19.648217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.647348    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerStarted","Data":"c288a5cab0b840d3240239dca9c6c726ae9287b5430410b6645574e69a97ec4c"}
Nov 02 18:00:19.664940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.663235    5042 generic.go:356] "Generic (PLEG): container finished" podID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerID="af07338de74b4bf65a704da6f99a854b1b49fd6138b7847c153698b192b72126" exitCode=60
Nov 02 18:00:19.664940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.663362    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerDied","Data":"af07338de74b4bf65a704da6f99a854b1b49fd6138b7847c153698b192b72126"}
Nov 02 18:00:19.664940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.663476    5042 scope.go:122] "RemoveContainer" containerID="c758ab08051c9fa3cdb258a96d8fdbbff9e9ad6ef2124923ec1907b658e624ff"
Nov 02 18:00:19.664940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.663965    5042 scope.go:122] "RemoveContainer" containerID="af07338de74b4bf65a704da6f99a854b1b49fd6138b7847c153698b192b72126"
Nov 02 18:00:19.664940 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:19.664457    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26_odh-ai-gateway-infra(1f8c0fbc-055c-4516-bfe3-5efbdeb91a06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06"
Nov 02 18:00:19.736089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.733095    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"]
Nov 02 18:00:19.816979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:19.814639    5042 scope.go:122] "RemoveContainer" containerID="3e47b0bb15393de8783e7740517182a292eb045aab1908fd542e205dcf543ddc"
Nov 02 18:00:20.051077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.051044    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"]
Nov 02 18:00:20.082622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.082591    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"]
Nov 02 18:00:20.096622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.096589    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894040-8s4ww"]
Nov 02 18:00:20.120076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.120049    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:20.195942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.184603    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z\" (UniqueName: \"kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z\") pod \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") "
Nov 02 18:00:20.206093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.206047    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca\") pod \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\" (UID: \"b7733536-09f3-4ed5-bbec-48eaea5e05e4\") "
Nov 02 18:00:20.208670 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.208621    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca" pod "b7733536-09f3-4ed5-bbec-48eaea5e05e4" (UID: "b7733536-09f3-4ed5-bbec-48eaea5e05e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.226992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.225761    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z" pod "b7733536-09f3-4ed5-bbec-48eaea5e05e4" (UID: "b7733536-09f3-4ed5-bbec-48eaea5e05e4"). InnerVolumeSpecName "kube-api-access-hnz5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.283271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.283239    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:20.315925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.315893    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7733536-09f3-4ed5-bbec-48eaea5e05e4-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.318487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.318449    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnz5z\" (UniqueName: \"kubernetes.io/projected/b7733536-09f3-4ed5-bbec-48eaea5e05e4-kube-api-access-hnz5z\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.335195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.334599    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:20.339492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.339467    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:20.341920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.341833    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:20.348756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.348700    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.394647    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"]
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395713    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395730    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395750    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395758    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395772    5042 cpu_manager.go:414] "containerMap: removing container" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395779    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395791    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395799    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395815    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395822    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395857    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395864    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395873    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395879    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395891    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.395890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395899    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395909    5042 cpu_manager.go:414] "containerMap: removing container" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395916    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395934    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395940    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395956    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395962    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395973    5042 cpu_manager.go:414] "containerMap: removing container" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395979    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.395995    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396002    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396013    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396020    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396041    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396048    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396056    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396062    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396074    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396080    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396105    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396111    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396310    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396325    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396334    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396344    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396353    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396363    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396374    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396386    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396397    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396406    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396416    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396428    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396436    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396445    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396454    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:20.396929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.396464    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:20.414135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.401236    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.423289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.421717    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca\") pod \"1e421df1-dfed-4d34-9ad6-6692ffa48115\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") "
Nov 02 18:00:20.423289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.421943    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4\" (UniqueName: \"kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4\") pod \"1e421df1-dfed-4d34-9ad6-6692ffa48115\" (UID: \"1e421df1-dfed-4d34-9ad6-6692ffa48115\") "
Nov 02 18:00:20.423289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.422070    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca\") pod \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") "
Nov 02 18:00:20.423289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.422255    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td\" (UniqueName: \"kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td\") pod \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\" (UID: \"c9f1b851-c125-41b3-81ae-3cbc1ec6e468\") "
Nov 02 18:00:20.424880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.424818    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca" pod "1e421df1-dfed-4d34-9ad6-6692ffa48115" (UID: "1e421df1-dfed-4d34-9ad6-6692ffa48115"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.425780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.425760    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca" pod "c9f1b851-c125-41b3-81ae-3cbc1ec6e468" (UID: "c9f1b851-c125-41b3-81ae-3cbc1ec6e468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.434762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.434145    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4" pod "1e421df1-dfed-4d34-9ad6-6692ffa48115" (UID: "1e421df1-dfed-4d34-9ad6-6692ffa48115"). InnerVolumeSpecName "kube-api-access-mtvk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.435702 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.435652    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"]
Nov 02 18:00:20.437431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.437381    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td" pod "c9f1b851-c125-41b3-81ae-3cbc1ec6e468" (UID: "c9f1b851-c125-41b3-81ae-3cbc1ec6e468"). InnerVolumeSpecName "kube-api-access-wk7td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.518974 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.506449    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-72f605-openshift-default-dockercfg-st7hd\""
Nov 02 18:00:20.585923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.525421    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca\") pod \"9290f100-625e-45d0-b4ec-216838391fbd\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") "
Nov 02 18:00:20.907413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.907383    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48\" (UniqueName: \"kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48\") pod \"255db28e-64ea-4f77-83d3-6a8930bd00f3\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") "
Nov 02 18:00:20.908122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.908097    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt\" (UniqueName: \"kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt\") pod \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") "
Nov 02 18:00:20.908457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.908437    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca\") pod \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\" (UID: \"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2\") "
Nov 02 18:00:20.908612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.908596    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj\" (UniqueName: \"kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj\") pod \"9290f100-625e-45d0-b4ec-216838391fbd\" (UID: \"9290f100-625e-45d0-b4ec-216838391fbd\") "
Nov 02 18:00:20.908755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.908742    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca\") pod \"255db28e-64ea-4f77-83d3-6a8930bd00f3\" (UID: \"255db28e-64ea-4f77-83d3-6a8930bd00f3\") "
Nov 02 18:00:20.909213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.909187    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.909387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.909370    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.909522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.909507    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.909627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.909612    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.913860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.909792    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.924211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.924175    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.924453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.924432    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.924791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.924771    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25j4\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.924948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.924932    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:20.925295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.925279    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7td\" (UniqueName: \"kubernetes.io/projected/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-kube-api-access-wk7td\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.925411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.925397    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e421df1-dfed-4d34-9ad6-6692ffa48115-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.930929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.930901    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtvk4\" (UniqueName: \"kubernetes.io/projected/1e421df1-dfed-4d34-9ad6-6692ffa48115-kube-api-access-mtvk4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.931105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.931091    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9f1b851-c125-41b3-81ae-3cbc1ec6e468-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:20.944355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.562253    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca" pod "9290f100-625e-45d0-b4ec-216838391fbd" (UID: "9290f100-625e-45d0-b4ec-216838391fbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.944679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.921172    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca" pod "255db28e-64ea-4f77-83d3-6a8930bd00f3" (UID: "255db28e-64ea-4f77-83d3-6a8930bd00f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.949249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.949204    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca" pod "bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" (UID: "bc955a92-f1af-4fd7-bbb4-c1a53691a7f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:20.950306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.950275    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt" pod "bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" (UID: "bc955a92-f1af-4fd7-bbb4-c1a53691a7f2"). InnerVolumeSpecName "kube-api-access-cvzgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.952537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.952495    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj" pod "9290f100-625e-45d0-b4ec-216838391fbd" (UID: "9290f100-625e-45d0-b4ec-216838391fbd"). InnerVolumeSpecName "kube-api-access-skpnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.952985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.952956    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48" pod "255db28e-64ea-4f77-83d3-6a8930bd00f3" (UID: "255db28e-64ea-4f77-83d3-6a8930bd00f3"). InnerVolumeSpecName "kube-api-access-8xr48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:20.957131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.957099    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2" event={"ID":"c9f1b851-c125-41b3-81ae-3cbc1ec6e468","Type":"ContainerDied","Data":"274a4647ef4ecbe28fe9294c4a27c358dd2e05de278aaf2149557f135d7bfd30"}
Nov 02 18:00:20.957232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.957174    5042 scope.go:122] "RemoveContainer" containerID="f4e99375c7a8be621ae9f7b539205564e11b9d7c4082299f100e4ce0ef8286fb"
Nov 02 18:00:20.957298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.957284    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"
Nov 02 18:00:20.961928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.961903    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"]
Nov 02 18:00:20.970085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.967306    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz" event={"ID":"1e421df1-dfed-4d34-9ad6-6692ffa48115","Type":"ContainerDied","Data":"633881ee0b030fb06001408f2e11ed2d742ed5d01d846b84e41e9792865f9eb2"}
Nov 02 18:00:20.970085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.967385    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"
Nov 02 18:00:20.979228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.979203    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"
Nov 02 18:00:20.979414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.979388    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d" event={"ID":"bc955a92-f1af-4fd7-bbb4-c1a53691a7f2","Type":"ContainerDied","Data":"05895fe1b5b900e63a0074e2d637c355422cfff5695990ee6d9e02826a1e1c09"}
Nov 02 18:00:20.982181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.982149    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc" event={"ID":"b7733536-09f3-4ed5-bbec-48eaea5e05e4","Type":"ContainerDied","Data":"73c628318d6905eec381cd8c8f57edf8ce95605e4dec0ff76a4265fd9711bc88"}
Nov 02 18:00:20.982536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.982516    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"
Nov 02 18:00:20.996053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.994357    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp" event={"ID":"255db28e-64ea-4f77-83d3-6a8930bd00f3","Type":"ContainerDied","Data":"bba5cfea375358dd450cc7aa61a86743e016c339fbbd418ddc15fed33db9fe14"}
Nov 02 18:00:20.996053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:20.994470    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"
Nov 02 18:00:21.008900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.008713    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb" event={"ID":"9290f100-625e-45d0-b4ec-216838391fbd","Type":"ContainerDied","Data":"b7b94cb3a4180b4ade03170af3e891f9b530b704260101678d8a53925c45a3de"}
Nov 02 18:00:21.008900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.008833    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"
Nov 02 18:00:21.018161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.018132    5042 scope.go:122] "RemoveContainer" containerID="b9946e629deda742604e6d43f6f1a933e674723f6080e0eb243ac13b4d02798a"
Nov 02 18:00:21.023640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.022428    5042 generic.go:356] "Generic (PLEG): container finished" podID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerID="c288a5cab0b840d3240239dca9c6c726ae9287b5430410b6645574e69a97ec4c" exitCode=60
Nov 02 18:00:21.023640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.022519    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerDied","Data":"c288a5cab0b840d3240239dca9c6c726ae9287b5430410b6645574e69a97ec4c"}
Nov 02 18:00:21.033620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.033584    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.033762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.033655    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.033831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.033813    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25j4\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.034718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.034689    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.034895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.034879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.034970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.034905    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.034970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.034945    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.035095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.034968    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.035095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035037    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.035240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035222    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xr48\" (UniqueName: \"kubernetes.io/projected/255db28e-64ea-4f77-83d3-6a8930bd00f3-kube-api-access-8xr48\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.035314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035246    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzgt\" (UniqueName: \"kubernetes.io/projected/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-kube-api-access-cvzgt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.035314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035262    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.035314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035275    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skpnj\" (UniqueName: \"kubernetes.io/projected/9290f100-625e-45d0-b4ec-216838391fbd-kube-api-access-skpnj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.035314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035288    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/255db28e-64ea-4f77-83d3-6a8930bd00f3-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.035314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.035304    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9290f100-625e-45d0-b4ec-216838391fbd-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.039568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.038661    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.039568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.039400    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.039568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.039548    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.040172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.040124    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.041811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.040598    5042 scope.go:122] "RemoveContainer" containerID="d802dfce44b243085ad7c9007064495726b97ade08684ef2ed267c65e77c0160"
Nov 02 18:00:21.045268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.045238    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"]
Nov 02 18:00:21.046291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.046231    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.049337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.049310    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314c8721-gpk8x"]
Nov 02 18:00:21.059624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.058464    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.070964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.063638    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25j4\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.070964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.070869    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.080289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.080223    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"]
Nov 02 18:00:21.098080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.091050    5042 scope.go:122] "RemoveContainer" containerID="cfd2906e5b80985625eb32bd9e843ef7d805c97a6584fe2f28e211e213a5d574"
Nov 02 18:00:21.105722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.105601    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-77f103-29894040-7svrz"]
Nov 02 18:00:21.110473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.110427    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy\") pod \"e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") " pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.120764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.120732    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"]
Nov 02 18:00:21.129923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.129786    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-a37cf0-29894040-mzhr2"]
Nov 02 18:00:21.260861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.245118    5042 scope.go:122] "RemoveContainer" containerID="740dbc1e4cf265e91727cf216364280d06d65a30bfd64e4678ea5b421d6d7dd6"
Nov 02 18:00:21.299164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.298860    5042 scope.go:122] "RemoveContainer" containerID="191c385a66a5a319143805188c5aa0f830655179e3b9728c8e8f751685728a7d"
Nov 02 18:00:21.327865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.323070    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"]
Nov 02 18:00:21.327865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.324799    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894040-fslvp"]
Nov 02 18:00:21.327865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.326419    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:21.353650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.349377    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"]
Nov 02 18:00:21.353853 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.353697    5042 scope.go:122] "RemoveContainer" containerID="7ccdaca4c6432dd7886cbd5485f729e3c09587b7a26db0c97b2cae524850e330"
Nov 02 18:00:21.371796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.371768    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894040-9hg6d"]
Nov 02 18:00:21.387960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.387895    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:21.392941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.392911    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"]
Nov 02 18:00:21.418975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.418928    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b1a788-29894040-k49sb"]
Nov 02 18:00:21.428191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.428145    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"]
Nov 02 18:00:21.428466 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.428447    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894040-f82wc"]
Nov 02 18:00:21.495826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.495790    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc\" (UniqueName: \"kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc\") pod \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") "
Nov 02 18:00:21.496037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.495901    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca\") pod \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\" (UID: \"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06\") "
Nov 02 18:00:21.501558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.501475    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca" pod "1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" (UID: "1f8c0fbc-055c-4516-bfe3-5efbdeb91a06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:21.501948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.501898    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc" pod "1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" (UID: "1f8c0fbc-055c-4516-bfe3-5efbdeb91a06"). InnerVolumeSpecName "kube-api-access-hqqhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:21.543342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.542995    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:21.555746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.555709    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" path="/var/lib/kubelet/pods/1e421df1-dfed-4d34-9ad6-6692ffa48115/volumes"
Nov 02 18:00:21.556781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.556752    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24432ad9-44d9-4ae0-95bb-6515a2d97b9e" path="/var/lib/kubelet/pods/24432ad9-44d9-4ae0-95bb-6515a2d97b9e/volumes"
Nov 02 18:00:21.558261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.558136    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" path="/var/lib/kubelet/pods/255db28e-64ea-4f77-83d3-6a8930bd00f3/volumes"
Nov 02 18:00:21.560270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.560219    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9290f100-625e-45d0-b4ec-216838391fbd" path="/var/lib/kubelet/pods/9290f100-625e-45d0-b4ec-216838391fbd/volumes"
Nov 02 18:00:21.561720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.561703    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" path="/var/lib/kubelet/pods/b7733536-09f3-4ed5-bbec-48eaea5e05e4/volumes"
Nov 02 18:00:21.562876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.562855    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" path="/var/lib/kubelet/pods/bc955a92-f1af-4fd7-bbb4-c1a53691a7f2/volumes"
Nov 02 18:00:21.563531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.563511    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f1b851-c125-41b3-81ae-3cbc1ec6e468" path="/var/lib/kubelet/pods/c9f1b851-c125-41b3-81ae-3cbc1ec6e468/volumes"
Nov 02 18:00:21.563652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.563571    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:21.564340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.564319    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" path="/var/lib/kubelet/pods/e009d895-ad21-4dc3-ba59-3337e595ed85/volumes"
Nov 02 18:00:21.597689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.597517    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9\" (UniqueName: \"kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9\") pod \"802a456f-0275-4373-9428-e5e62e98b942\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") "
Nov 02 18:00:21.597883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.597713    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca\") pod \"802a456f-0275-4373-9428-e5e62e98b942\" (UID: \"802a456f-0275-4373-9428-e5e62e98b942\") "
Nov 02 18:00:21.597883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.597743    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf\" (UniqueName: \"kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf\") pod \"96d92a33-9ea9-47c7-8fb1-35804aad17af\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") "
Nov 02 18:00:21.598704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.598029    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca\") pod \"96d92a33-9ea9-47c7-8fb1-35804aad17af\" (UID: \"96d92a33-9ea9-47c7-8fb1-35804aad17af\") "
Nov 02 18:00:21.600338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.600309    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.600470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.600456    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqhc\" (UniqueName: \"kubernetes.io/projected/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06-kube-api-access-hqqhc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.601819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.601777    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca" pod "802a456f-0275-4373-9428-e5e62e98b942" (UID: "802a456f-0275-4373-9428-e5e62e98b942"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:21.603030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.602910    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca" pod "96d92a33-9ea9-47c7-8fb1-35804aad17af" (UID: "96d92a33-9ea9-47c7-8fb1-35804aad17af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:21.606032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.606005    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf" pod "96d92a33-9ea9-47c7-8fb1-35804aad17af" (UID: "96d92a33-9ea9-47c7-8fb1-35804aad17af"). InnerVolumeSpecName "kube-api-access-7twdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:21.607352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.607327    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9" pod "802a456f-0275-4373-9428-e5e62e98b942" (UID: "802a456f-0275-4373-9428-e5e62e98b942"). InnerVolumeSpecName "kube-api-access-6r8l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:21.694166 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:21.694121    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04ba0c7_e939_4825_81be_080c7b4139d9.slice/crio-15aea4b72350e16665c16842149d97d0b05e53021c802f2f7a91c3076f26a911 WatchSource:0}: Error finding container 15aea4b72350e16665c16842149d97d0b05e53021c802f2f7a91c3076f26a911: Status 404 returned error can't find the container with id 15aea4b72350e16665c16842149d97d0b05e53021c802f2f7a91c3076f26a911
Nov 02 18:00:21.694382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.694361    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"]
Nov 02 18:00:21.697213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.697181    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:21.697316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.697247    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:21.697316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.697277    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:21.702194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.702172    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d92a33-9ea9-47c7-8fb1-35804aad17af-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.702289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.702200    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8l9\" (UniqueName: \"kubernetes.io/projected/802a456f-0275-4373-9428-e5e62e98b942-kube-api-access-6r8l9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.702289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.702219    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/802a456f-0275-4373-9428-e5e62e98b942-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:21.702289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:21.702232    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twdf\" (UniqueName: \"kubernetes.io/projected/96d92a33-9ea9-47c7-8fb1-35804aad17af-kube-api-access-7twdf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:22.036907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.036763    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"
Nov 02 18:00:22.036907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.036788    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg" event={"ID":"802a456f-0275-4373-9428-e5e62e98b942","Type":"ContainerDied","Data":"762fb96faebfbad19d8beddc6cbcd4f8a27b2295ea6a79ec882081a559631e73"}
Nov 02 18:00:22.036907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.036877    5042 scope.go:122] "RemoveContainer" containerID="025b493787e635fdcd7d3ddddc7362a36061341081458a2ba03eed86b32421d4"
Nov 02 18:00:22.039457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.039398    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"
Nov 02 18:00:22.039457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.039449    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt" event={"ID":"96d92a33-9ea9-47c7-8fb1-35804aad17af","Type":"ContainerDied","Data":"603580e4dc83305d625b13899fc120b78bbe8bc304aa11c63af41e66b5011e77"}
Nov 02 18:00:22.048687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.048624    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26" event={"ID":"1f8c0fbc-055c-4516-bfe3-5efbdeb91a06","Type":"ContainerDied","Data":"0f849a2785dcd7d7c1963f82cd1416c50b8cafcafb2d546c872cdea809c01448"}
Nov 02 18:00:22.048833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.048772    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"
Nov 02 18:00:22.054577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.054544    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" event={"ID":"c04ba0c7-e939-4825-81be-080c7b4139d9","Type":"ContainerStarted","Data":"0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"}
Nov 02 18:00:22.054719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.054602    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" event={"ID":"c04ba0c7-e939-4825-81be-080c7b4139d9","Type":"ContainerStarted","Data":"15aea4b72350e16665c16842149d97d0b05e53021c802f2f7a91c3076f26a911"}
Nov 02 18:00:22.069860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.069814    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"]
Nov 02 18:00:22.071724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.069904    5042 scope.go:122] "RemoveContainer" containerID="c288a5cab0b840d3240239dca9c6c726ae9287b5430410b6645574e69a97ec4c"
Nov 02 18:00:22.079192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.079158    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894040-pzpkg"]
Nov 02 18:00:22.087998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.087975    5042 scope.go:122] "RemoveContainer" containerID="af07338de74b4bf65a704da6f99a854b1b49fd6138b7847c153698b192b72126"
Nov 02 18:00:22.119968 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.119911    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podStartSLOduration=2.119891506 podStartE2EDuration="2.119891506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:22.097857005 +0000 UTC m=+1634.677266382" watchObservedRunningTime="2026-11-02 18:00:22.119891506 +0000 UTC m=+1634.699300892"
Nov 02 18:00:22.122308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.121942    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"]
Nov 02 18:00:22.129194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.129161    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894040-z8d26"]
Nov 02 18:00:22.145212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.145180    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"]
Nov 02 18:00:22.149603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.149577    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894040-trnmt"]
Nov 02 18:00:22.335920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.333699    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:22.335920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.335243    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:00:22.335920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.335300    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:00:22.362096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.360557    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"]
Nov 02 18:00:22.362096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.360830    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy" containerID="cri-o://1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7" gracePeriod=30
Nov 02 18:00:22.368274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.367742    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"]
Nov 02 18:00:22.368274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:22.367965    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy" containerID="cri-o://f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9" gracePeriod=30
Nov 02 18:00:23.337157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:23.337120    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:23.546699 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:23.546664    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" path="/var/lib/kubelet/pods/1f8c0fbc-055c-4516-bfe3-5efbdeb91a06/volumes"
Nov 02 18:00:23.547229 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:23.547213    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="802a456f-0275-4373-9428-e5e62e98b942" path="/var/lib/kubelet/pods/802a456f-0275-4373-9428-e5e62e98b942/volumes"
Nov 02 18:00:23.547624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:23.547611    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" path="/var/lib/kubelet/pods/96d92a33-9ea9-47c7-8fb1-35804aad17af/volumes"
Nov 02 18:00:24.076435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:24.076397    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:24.077696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:24.077673    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:00:25.026770 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.026731    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"]
Nov 02 18:00:25.031223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.031196    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33b05595-556xm"]
Nov 02 18:00:25.551745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.551706    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c7a58d-81cd-4b25-a3fa-457afef7fe3d" path="/var/lib/kubelet/pods/94c7a58d-81cd-4b25-a3fa-457afef7fe3d/volumes"
Nov 02 18:00:25.608507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.608468    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"]
Nov 02 18:00:25.609324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609298    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.609324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609324    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609341    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609348    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7733536-09f3-4ed5-bbec-48eaea5e05e4" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609365    5042 cpu_manager.go:414] "containerMap: removing container" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609372    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609385    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609391    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609431    5042 cpu_manager.go:414] "containerMap: removing container" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609438    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609457    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609465    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609474    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609480    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609505    5042 cpu_manager.go:414] "containerMap: removing container" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609512    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609524    5042 cpu_manager.go:414] "containerMap: removing container" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.609537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609531    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609572    5042 cpu_manager.go:414] "containerMap: removing container" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609580    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609591    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609599    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609793    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e009d895-ad21-4dc3-ba59-3337e595ed85" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609809    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609818    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609829    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="255db28e-64ea-4f77-83d3-6a8930bd00f3" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609860    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609870    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609881    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e421df1-dfed-4d34-9ad6-6692ffa48115" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609895    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9290f100-625e-45d0-b4ec-216838391fbd" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609906    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc955a92-f1af-4fd7-bbb4-c1a53691a7f2" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609915    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609927    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.610401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.609938    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.619589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.619551    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.625724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.625693    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-72f605\""
Nov 02 18:00:25.627533 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.627504    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"]
Nov 02 18:00:25.628727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.628699    5042 cpu_manager.go:414] "containerMap: removing container" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.628914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.628729    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d92a33-9ea9-47c7-8fb1-35804aad17af" containerName="cleanup"
Nov 02 18:00:25.628988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.628917    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f8c0fbc-055c-4516-bfe3-5efbdeb91a06" containerName="cleanup"
Nov 02 18:00:25.628988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.628940    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="802a456f-0275-4373-9428-e5e62e98b942" containerName="cleanup"
Nov 02 18:00:25.637034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.635977    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.637923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.637816    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"]
Nov 02 18:00:25.642425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.641505    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-72f605-dockercfg-vww4n\""
Nov 02 18:00:25.642425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.642364    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"]
Nov 02 18:00:25.721376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.721328    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskxf\" (UniqueName: \"kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf\") pod \"payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.721556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.721507    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.721782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.721755    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44sfs\" (UniqueName: \"kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs\") pod \"payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.722105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.722074    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.838922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.825112    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44sfs\" (UniqueName: \"kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs\") pod \"payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.838922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.825306    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.838922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.825471    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskxf\" (UniqueName: \"kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf\") pod \"payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.838922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.825537    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.854873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.851060    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.854873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.853727    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.877522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.877451    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskxf\" (UniqueName: \"kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf\") pod \"payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.925496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.909497    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44sfs\" (UniqueName: \"kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs\") pod \"payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:25.946783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.946629    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:25.967023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:25.966771    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:26.897081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:26.897040    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"]
Nov 02 18:00:26.906562 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:26.906507    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"]
Nov 02 18:00:27.122889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.122731    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6" event={"ID":"1fe5753a-dd42-4a3b-8711-cf8868a0f61f","Type":"ContainerStarted","Data":"43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7"}
Nov 02 18:00:27.122889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.122787    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6" event={"ID":"1fe5753a-dd42-4a3b-8711-cf8868a0f61f","Type":"ContainerStarted","Data":"8a1782c06056db2f4bae2d6603a7e04bc014681d93fd7d94ae829b5949e8e0fd"}
Nov 02 18:00:27.123340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.123311    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:27.125356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.125323    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr" event={"ID":"b82e6528-8aa6-48dc-8758-7d13c89902cf","Type":"ContainerStarted","Data":"cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594"}
Nov 02 18:00:27.126053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.125377    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr" event={"ID":"b82e6528-8aa6-48dc-8758-7d13c89902cf","Type":"ContainerStarted","Data":"87ebc70e6f5d90abfb34cc7296917fcd306f44c711e1d783c91674e7394d33d6"}
Nov 02 18:00:27.126053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.125436    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:27.145531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.145479    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6" podStartSLOduration=2.145464637 podStartE2EDuration="2.145464637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:27.14040521 +0000 UTC m=+1639.719814608" watchObservedRunningTime="2026-11-02 18:00:27.145464637 +0000 UTC m=+1639.724874023"
Nov 02 18:00:27.158455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.158393    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr" podStartSLOduration=2.158375932 podStartE2EDuration="2.158375932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:27.157136195 +0000 UTC m=+1639.736545581" watchObservedRunningTime="2026-11-02 18:00:27.158375932 +0000 UTC m=+1639.737785318"
Nov 02 18:00:27.687653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.687618    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 18:00:27.818561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.818525    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.823419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.823388    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.824071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.824050    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.828637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.828613    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.830878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.830857    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.831344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.831326    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.831942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.831924    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.832428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.832410    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.832869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.832823    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token\") pod \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\" (UID: \"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9\") "
Nov 02 18:00:27.834150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.830730    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 18:00:27.834150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.820467    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:27.834150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.823894    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:27.834150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.829972    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:27.834150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.832337    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:27.834150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.832703    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:27.880100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.859726    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:27.880100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.859766    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:27.880100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.859781    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:27.880100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.859795    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:27.880100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.859808    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:27.880100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.861571    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.966423    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.966487    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.966570    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.966777    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.966811    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.966977    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.967147    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.967204    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:27.967865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:27.967295    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket\") pod \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\" (UID: \"fa2ee4bf-7406-4fe9-bfc1-569e0192d292\") "
Nov 02 18:00:28.027525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.017945    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:28.028426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.028381    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:28.032038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.029929    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:28.032038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.030265    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:28.069866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.039985    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.069866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.040016    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.069866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.040029    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.069866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.040042    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.069866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.040055    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.112318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.111639    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:28.119675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.119615    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151122    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151123    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151185    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151317    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151403    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "kube-api-access-pwvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151532    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "kube-api-access-x28hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:28.152114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.151893    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy" pod "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" (UID: "8f71cf54-6fae-4b35-8f0b-0a6ece81acb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:28.154828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.154780    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy" pod "fa2ee4bf-7406-4fe9-bfc1-569e0192d292" (UID: "fa2ee4bf-7406-4fe9-bfc1-569e0192d292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:28.196374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.195959    5042 generic.go:356] "Generic (PLEG): container finished" podID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerID="1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7" exitCode=0
Nov 02 18:00:28.196374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.196209    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" event={"ID":"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9","Type":"ContainerDied","Data":"1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7"}
Nov 02 18:00:28.196919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.196677    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7" event={"ID":"8f71cf54-6fae-4b35-8f0b-0a6ece81acb9","Type":"ContainerDied","Data":"8946a90b4d46949f4f8799fd275a802ea7095c1aafc738b75cfd9742fe910aae"}
Nov 02 18:00:28.196919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.196729    5042 scope.go:122] "RemoveContainer" containerID="1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7"
Nov 02 18:00:28.200097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.197223    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"
Nov 02 18:00:28.206576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.206544    5042 generic.go:356] "Generic (PLEG): container finished" podID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerID="f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9" exitCode=0
Nov 02 18:00:28.212181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.212155    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"
Nov 02 18:00:28.214864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.214159    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" event={"ID":"fa2ee4bf-7406-4fe9-bfc1-569e0192d292","Type":"ContainerDied","Data":"f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9"}
Nov 02 18:00:28.214864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.214210    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg" event={"ID":"fa2ee4bf-7406-4fe9-bfc1-569e0192d292","Type":"ContainerDied","Data":"2bedc43120133bdc9bd7b6894c8ab72a6b829cb13afb5d3e5f2996ece6717053"}
Nov 02 18:00:28.253403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.253371    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.253615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.253595    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvzf\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-kube-api-access-pwvzf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.253750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.253733    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.253922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.253906    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.254050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.254035    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28hv\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-kube-api-access-x28hv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.254146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.254133    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa2ee4bf-7406-4fe9-bfc1-569e0192d292-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:28.271116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.271084    5042 scope.go:122] "RemoveContainer" containerID="1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7"
Nov 02 18:00:28.271663 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:28.271627    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7\": container with ID starting with 1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7 not found: ID does not exist" containerID="1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7"
Nov 02 18:00:28.271782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.271676    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7"} err="failed to get container status \"1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7\": rpc error: code = NotFound desc = could not find container \"1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7\": container with ID starting with 1c806019c8ef72dd9931bc99ec352fed1711608e1097238cf67329a8cdd7ebe7 not found: ID does not exist"
Nov 02 18:00:28.271782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.271703    5042 scope.go:122] "RemoveContainer" containerID="f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9"
Nov 02 18:00:28.275884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.275787    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"]
Nov 02 18:00:28.293441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.293405    5042 scope.go:122] "RemoveContainer" containerID="f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9"
Nov 02 18:00:28.293947 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:28.293917    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9\": container with ID starting with f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9 not found: ID does not exist" containerID="f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9"
Nov 02 18:00:28.294059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.293960    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9"} err="failed to get container status \"f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9\": rpc error: code = NotFound desc = could not find container \"f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9\": container with ID starting with f205417fed4ad54976a6829fa37cf93d64c7c89d8521a5f9ddfb032aca9ed4c9 not found: ID does not exist"
Nov 02 18:00:28.300220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.300189    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kxhd7"]
Nov 02 18:00:28.314309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.310656    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"]
Nov 02 18:00:28.318935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:28.318901    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-5ppgg"]
Nov 02 18:00:29.546720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:29.546683    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" path="/var/lib/kubelet/pods/8f71cf54-6fae-4b35-8f0b-0a6ece81acb9/volumes"
Nov 02 18:00:29.547446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:29.547424    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" path="/var/lib/kubelet/pods/fa2ee4bf-7406-4fe9-bfc1-569e0192d292/volumes"
Nov 02 18:00:30.025568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.025533    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"]
Nov 02 18:00:30.038761 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.038720    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4db47eb-znxsw"]
Nov 02 18:00:30.909200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.909157    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"]
Nov 02 18:00:30.909981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.909964    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy"
Nov 02 18:00:30.909981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.909983    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy"
Nov 02 18:00:30.910095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.910001    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy"
Nov 02 18:00:30.910095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.910006    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy"
Nov 02 18:00:30.910162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.910143    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f71cf54-6fae-4b35-8f0b-0a6ece81acb9" containerName="istio-proxy"
Nov 02 18:00:30.910162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.910152    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2ee4bf-7406-4fe9-bfc1-569e0192d292" containerName="istio-proxy"
Nov 02 18:00:30.939356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.939319    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:30.946550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.946517    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:00:30.953886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.947122    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:00:30.955061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.950040    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:00:30.959328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.950178    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:00:30.962091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:30.962059    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"]
Nov 02 18:00:31.053249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.053218    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fc88\" (UniqueName: \"kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.053539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.053511    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.053802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.053778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.154983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.154938    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.155223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.155120    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.155223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.155203    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fc88\" (UniqueName: \"kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.155901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.155876    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.157934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.157907    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.163923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.163796    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fc88\" (UniqueName: \"kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88\") pod \"maas-api-revoke-sub-a454a017-t8hlx\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.304772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.304734    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:31.547476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.547443    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05ea8853-9e7c-487a-ac09-275f3a589b54" path="/var/lib/kubelet/pods/05ea8853-9e7c-487a-ac09-275f3a589b54/volumes"
Nov 02 18:00:31.654866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:31.651343    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"]
Nov 02 18:00:32.259997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:32.259883    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx" event={"ID":"4eb22fbd-150b-48ba-9e45-f4a1e792df8c","Type":"ContainerStarted","Data":"78a10e1d5bfecf95c912ee2bbb7b7392cbbf3c9e7b7dbb32af6b19ad6fdf6f21"}
Nov 02 18:00:32.259997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:32.259950    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx" event={"ID":"4eb22fbd-150b-48ba-9e45-f4a1e792df8c","Type":"ContainerStarted","Data":"92e1daec19a5745020c9005ec20e817cd819c4e62edfdab6124654721583b0e1"}
Nov 02 18:00:32.327862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:32.322249    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx" podStartSLOduration=2.322222764 podStartE2EDuration="2.322222764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:32.319015083 +0000 UTC m=+1644.898424470" watchObservedRunningTime="2026-11-02 18:00:32.322222764 +0000 UTC m=+1644.901632169"
Nov 02 18:00:33.284581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:33.284544    5042 generic.go:356] "Generic (PLEG): container finished" podID="4eb22fbd-150b-48ba-9e45-f4a1e792df8c" containerID="78a10e1d5bfecf95c912ee2bbb7b7392cbbf3c9e7b7dbb32af6b19ad6fdf6f21" exitCode=0
Nov 02 18:00:33.285101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:33.284633    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx" event={"ID":"4eb22fbd-150b-48ba-9e45-f4a1e792df8c","Type":"ContainerDied","Data":"78a10e1d5bfecf95c912ee2bbb7b7392cbbf3c9e7b7dbb32af6b19ad6fdf6f21"}
Nov 02 18:00:34.588371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.588334    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:34.673484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.673441    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca\") pod \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") "
Nov 02 18:00:34.673681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.673543    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca\") pod \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") "
Nov 02 18:00:34.673681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.673612    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88\" (UniqueName: \"kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88\") pod \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\" (UID: \"4eb22fbd-150b-48ba-9e45-f4a1e792df8c\") "
Nov 02 18:00:34.674120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.674092    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca" pod "4eb22fbd-150b-48ba-9e45-f4a1e792df8c" (UID: "4eb22fbd-150b-48ba-9e45-f4a1e792df8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:34.675482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.675456    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:34.676709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.676680    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca" pod "4eb22fbd-150b-48ba-9e45-f4a1e792df8c" (UID: "4eb22fbd-150b-48ba-9e45-f4a1e792df8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:34.676709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.676684    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88" pod "4eb22fbd-150b-48ba-9e45-f4a1e792df8c" (UID: "4eb22fbd-150b-48ba-9e45-f4a1e792df8c"). InnerVolumeSpecName "kube-api-access-6fc88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:34.777249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.777210    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:34.777249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:34.777248    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fc88\" (UniqueName: \"kubernetes.io/projected/4eb22fbd-150b-48ba-9e45-f4a1e792df8c-kube-api-access-6fc88\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:35.033365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:35.033280    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"]
Nov 02 18:00:35.043627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:35.043590    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce41582-w75f4"]
Nov 02 18:00:35.305231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:35.305136    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"
Nov 02 18:00:35.305231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:35.305176    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx" event={"ID":"4eb22fbd-150b-48ba-9e45-f4a1e792df8c","Type":"ContainerDied","Data":"92e1daec19a5745020c9005ec20e817cd819c4e62edfdab6124654721583b0e1"}
Nov 02 18:00:35.305475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:35.305236    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92e1daec19a5745020c9005ec20e817cd819c4e62edfdab6124654721583b0e1"
Nov 02 18:00:35.549335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:35.549303    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f37c283-4680-451d-ae2e-1cc801ca0768" path="/var/lib/kubelet/pods/9f37c283-4680-451d-ae2e-1cc801ca0768/volumes"
Nov 02 18:00:38.027038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:38.027001    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"]
Nov 02 18:00:38.031440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:38.031413    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ae5731b-6nqx5"]
Nov 02 18:00:38.217655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:38.217626    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:00:38.217924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:38.217703    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:00:39.583689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:39.582276    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2389da41-ad7e-4674-90ce-79214e405ac0" path="/var/lib/kubelet/pods/2389da41-ad7e-4674-90ce-79214e405ac0/volumes"
Nov 02 18:00:41.145176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.145141    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"]
Nov 02 18:00:41.146039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.145899    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4eb22fbd-150b-48ba-9e45-f4a1e792df8c" containerName="revoke-keys"
Nov 02 18:00:41.146039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.145912    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb22fbd-150b-48ba-9e45-f4a1e792df8c" containerName="revoke-keys"
Nov 02 18:00:41.146153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.146049    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb22fbd-150b-48ba-9e45-f4a1e792df8c" containerName="revoke-keys"
Nov 02 18:00:41.162459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.162424    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.163824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.163799    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"]
Nov 02 18:00:41.167363 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.167336    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:00:41.167513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.167371    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:00:41.171887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.171860    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:00:41.172106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.172034    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:00:41.275809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.275756    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.276032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.275932    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.276139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.276118    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flz7w\" (UniqueName: \"kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.379723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.378281    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flz7w\" (UniqueName: \"kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.379723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.378544    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.379723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.378611    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.379723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.379508    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.382454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.382430    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.391166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.391119    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flz7w\" (UniqueName: \"kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w\") pod \"maas-api-revoke-sub-bbccfafd-cgprf\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.484391 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.484359    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:41.797395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:41.797361    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"]
Nov 02 18:00:41.802179 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:41.799634    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc5a570_9741_4e52_b5fa_8e1b89e243a4.slice/crio-bc120d94619de24b28acc83c197a618fae6480ff10bffeedd9d16d618c1f7788 WatchSource:0}: Error finding container bc120d94619de24b28acc83c197a618fae6480ff10bffeedd9d16d618c1f7788: Status 404 returned error can't find the container with id bc120d94619de24b28acc83c197a618fae6480ff10bffeedd9d16d618c1f7788
Nov 02 18:00:42.387310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:42.387202    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf" event={"ID":"7dc5a570-9741-4e52-b5fa-8e1b89e243a4","Type":"ContainerStarted","Data":"377849398f98709203f30eac2ddc849a2d548d8daab9f583ef216d5426ef0ae1"}
Nov 02 18:00:42.387310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:42.387273    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf" event={"ID":"7dc5a570-9741-4e52-b5fa-8e1b89e243a4","Type":"ContainerStarted","Data":"bc120d94619de24b28acc83c197a618fae6480ff10bffeedd9d16d618c1f7788"}
Nov 02 18:00:42.409881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:42.409794    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf" podStartSLOduration=1.409774717 podStartE2EDuration="1.409774717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:42.405669203 +0000 UTC m=+1654.985078622" watchObservedRunningTime="2026-11-02 18:00:42.409774717 +0000 UTC m=+1654.989184105"
Nov 02 18:00:43.408137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:43.408095    5042 generic.go:356] "Generic (PLEG): container finished" podID="7dc5a570-9741-4e52-b5fa-8e1b89e243a4" containerID="377849398f98709203f30eac2ddc849a2d548d8daab9f583ef216d5426ef0ae1" exitCode=0
Nov 02 18:00:43.408670 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:43.408444    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf" event={"ID":"7dc5a570-9741-4e52-b5fa-8e1b89e243a4","Type":"ContainerDied","Data":"377849398f98709203f30eac2ddc849a2d548d8daab9f583ef216d5426ef0ae1"}
Nov 02 18:00:44.611559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.611520    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:44.740656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.740620    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca\") pod \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") "
Nov 02 18:00:44.740893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.740829    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w\" (UniqueName: \"kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w\") pod \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") "
Nov 02 18:00:44.740961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.740944    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca\") pod \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\" (UID: \"7dc5a570-9741-4e52-b5fa-8e1b89e243a4\") "
Nov 02 18:00:44.741012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.740989    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca" pod "7dc5a570-9741-4e52-b5fa-8e1b89e243a4" (UID: "7dc5a570-9741-4e52-b5fa-8e1b89e243a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:44.741663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.741643    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:44.743163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.743131    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w" pod "7dc5a570-9741-4e52-b5fa-8e1b89e243a4" (UID: "7dc5a570-9741-4e52-b5fa-8e1b89e243a4"). InnerVolumeSpecName "kube-api-access-flz7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:44.743270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.743157    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca" pod "7dc5a570-9741-4e52-b5fa-8e1b89e243a4" (UID: "7dc5a570-9741-4e52-b5fa-8e1b89e243a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:44.843596 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.843560    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flz7w\" (UniqueName: \"kubernetes.io/projected/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-kube-api-access-flz7w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:44.843596 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:44.843593    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dc5a570-9741-4e52-b5fa-8e1b89e243a4-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:45.029449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:45.029356    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"]
Nov 02 18:00:45.038576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:45.038542    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c186922-b82pb"]
Nov 02 18:00:45.431073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:45.430989    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"
Nov 02 18:00:45.431073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:45.431026    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf" event={"ID":"7dc5a570-9741-4e52-b5fa-8e1b89e243a4","Type":"ContainerDied","Data":"bc120d94619de24b28acc83c197a618fae6480ff10bffeedd9d16d618c1f7788"}
Nov 02 18:00:45.431298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:45.431103    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc120d94619de24b28acc83c197a618fae6480ff10bffeedd9d16d618c1f7788"
Nov 02 18:00:45.593879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:45.593809    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc9f21ab-3e61-407a-b465-213b7658fd9e" path="/var/lib/kubelet/pods/cc9f21ab-3e61-407a-b465-213b7658fd9e/volumes"
Nov 02 18:00:48.034520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.034410    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"]
Nov 02 18:00:48.035558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.035532    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7dc5a570-9741-4e52-b5fa-8e1b89e243a4" containerName="revoke-keys"
Nov 02 18:00:48.035558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.035558    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc5a570-9741-4e52-b5fa-8e1b89e243a4" containerName="revoke-keys"
Nov 02 18:00:48.035884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.035821    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc5a570-9741-4e52-b5fa-8e1b89e243a4" containerName="revoke-keys"
Nov 02 18:00:48.040830 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.040802    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.048300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.048241    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:00:48.048581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.048552    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"]
Nov 02 18:00:48.049445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.049423    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:00:48.049612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.049594    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:00:48.049713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.049679    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:00:48.113219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.113185    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"]
Nov 02 18:00:48.119686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.118421    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.155048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.155012    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"]
Nov 02 18:00:48.204314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.204274    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvglj\" (UniqueName: \"kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.206625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.206588    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.207003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.206972    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.207108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.207073    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7h7w\" (UniqueName: \"kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.207167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.207139    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.207222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.207189    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.308610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.308511    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.308610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.308598    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7h7w\" (UniqueName: \"kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.308898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.308648    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.308898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.308687    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.309023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.308943    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvglj\" (UniqueName: \"kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.309097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.309081    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.309913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.309826    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.309913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.309831    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.311599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.311568    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.312001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.311975    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.321007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.320978    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvglj\" (UniqueName: \"kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj\") pod \"maas-api-revoke-sub-c7c2054d-ftbm9\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.321303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.321284    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7h7w\" (UniqueName: \"kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w\") pod \"maas-api-revoke-sub-220152a3-zjvqn\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.381692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.381649    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:48.471720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.468728    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:48.712513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:48.708485    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"]
Nov 02 18:00:48.719214 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:00:48.719157    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:00:48.726162 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:48.726128    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e16960_7229_4ea9_a864_71f4449345cc.slice/crio-d3b1026cff94903b33bd48508249cb3cfed485a9d0234830daecb6e97fc7909c WatchSource:0}: Error finding container d3b1026cff94903b33bd48508249cb3cfed485a9d0234830daecb6e97fc7909c: Status 404 returned error can't find the container with id d3b1026cff94903b33bd48508249cb3cfed485a9d0234830daecb6e97fc7909c
Nov 02 18:00:49.013235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.013198    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"]
Nov 02 18:00:49.014335 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:49.014307    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9b794_4f22_4e6d_a851_b5d5dda79e38.slice/crio-5714327053f3531a92b94d07b9867f3f81732be2c5c6b42316324a17aee35b47 WatchSource:0}: Error finding container 5714327053f3531a92b94d07b9867f3f81732be2c5c6b42316324a17aee35b47: Status 404 returned error can't find the container with id 5714327053f3531a92b94d07b9867f3f81732be2c5c6b42316324a17aee35b47
Nov 02 18:00:49.663940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.663815    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn" event={"ID":"d2f9b794-4f22-4e6d-a851-b5d5dda79e38","Type":"ContainerStarted","Data":"5b0e498c0f911f24950e5d552ad080fd376aa1cbe3eee392e39eacbf2979b648"}
Nov 02 18:00:49.663940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.663890    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn" event={"ID":"d2f9b794-4f22-4e6d-a851-b5d5dda79e38","Type":"ContainerStarted","Data":"5714327053f3531a92b94d07b9867f3f81732be2c5c6b42316324a17aee35b47"}
Nov 02 18:00:49.666074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.666043    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9" event={"ID":"b3e16960-7229-4ea9-a864-71f4449345cc","Type":"ContainerStarted","Data":"2872f80fc7350661ac30eb5eb900642a702b9c09606e7e0ee65ff918f842dd54"}
Nov 02 18:00:49.666218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.666098    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9" event={"ID":"b3e16960-7229-4ea9-a864-71f4449345cc","Type":"ContainerStarted","Data":"d3b1026cff94903b33bd48508249cb3cfed485a9d0234830daecb6e97fc7909c"}
Nov 02 18:00:49.684133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.684085    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn" podStartSLOduration=1.6840693 podStartE2EDuration="1.6840693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:49.680082779 +0000 UTC m=+1662.259492180" watchObservedRunningTime="2026-11-02 18:00:49.6840693 +0000 UTC m=+1662.263478686"
Nov 02 18:00:49.702238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:49.702177    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9" podStartSLOduration=1.7021624389999999 podStartE2EDuration="1.702162439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:49.69792547 +0000 UTC m=+1662.277334857" watchObservedRunningTime="2026-11-02 18:00:49.702162439 +0000 UTC m=+1662.281571826"
Nov 02 18:00:50.677297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:50.677207    5042 generic.go:356] "Generic (PLEG): container finished" podID="d2f9b794-4f22-4e6d-a851-b5d5dda79e38" containerID="5b0e498c0f911f24950e5d552ad080fd376aa1cbe3eee392e39eacbf2979b648" exitCode=0
Nov 02 18:00:50.677769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:50.677295    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn" event={"ID":"d2f9b794-4f22-4e6d-a851-b5d5dda79e38","Type":"ContainerDied","Data":"5b0e498c0f911f24950e5d552ad080fd376aa1cbe3eee392e39eacbf2979b648"}
Nov 02 18:00:50.679694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:50.679669    5042 generic.go:356] "Generic (PLEG): container finished" podID="b3e16960-7229-4ea9-a864-71f4449345cc" containerID="2872f80fc7350661ac30eb5eb900642a702b9c09606e7e0ee65ff918f842dd54" exitCode=0
Nov 02 18:00:50.679814 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:50.679753    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9" event={"ID":"b3e16960-7229-4ea9-a864-71f4449345cc","Type":"ContainerDied","Data":"2872f80fc7350661ac30eb5eb900642a702b9c09606e7e0ee65ff918f842dd54"}
Nov 02 18:00:51.696610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.696575    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"]
Nov 02 18:00:51.741503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.739364    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"]
Nov 02 18:00:51.741503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.739497    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.839575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.839526    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.840000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.839981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqtk\" (UniqueName: \"kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.840193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.840178    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.943328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.943255    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.943551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.943529    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqtk\" (UniqueName: \"kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.943701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.943685    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.944437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.944125    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.950267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.950218    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:51.954715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:51.954669    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqtk\" (UniqueName: \"kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk\") pod \"maas-api-revoke-sub-c7ad00fc-26w47\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:52.008217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.008186    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:52.011075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.011048    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:52.065582 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.065552    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:52.145923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.145887    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca\") pod \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") "
Nov 02 18:00:52.146140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.146025    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca\") pod \"b3e16960-7229-4ea9-a864-71f4449345cc\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") "
Nov 02 18:00:52.146211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.146142    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca\") pod \"b3e16960-7229-4ea9-a864-71f4449345cc\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") "
Nov 02 18:00:52.146211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.146171    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj\" (UniqueName: \"kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj\") pod \"b3e16960-7229-4ea9-a864-71f4449345cc\" (UID: \"b3e16960-7229-4ea9-a864-71f4449345cc\") "
Nov 02 18:00:52.146211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.146195    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w\" (UniqueName: \"kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w\") pod \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") "
Nov 02 18:00:52.146384 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.146273    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca\") pod \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\" (UID: \"d2f9b794-4f22-4e6d-a851-b5d5dda79e38\") "
Nov 02 18:00:52.147453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.147419    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca" pod "d2f9b794-4f22-4e6d-a851-b5d5dda79e38" (UID: "d2f9b794-4f22-4e6d-a851-b5d5dda79e38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:52.147938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.147891    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca" pod "b3e16960-7229-4ea9-a864-71f4449345cc" (UID: "b3e16960-7229-4ea9-a864-71f4449345cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:52.150333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.150298    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca" pod "d2f9b794-4f22-4e6d-a851-b5d5dda79e38" (UID: "d2f9b794-4f22-4e6d-a851-b5d5dda79e38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:52.154673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.154126    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj" pod "b3e16960-7229-4ea9-a864-71f4449345cc" (UID: "b3e16960-7229-4ea9-a864-71f4449345cc"). InnerVolumeSpecName "kube-api-access-kvglj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:52.154673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.154229    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca" pod "b3e16960-7229-4ea9-a864-71f4449345cc" (UID: "b3e16960-7229-4ea9-a864-71f4449345cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:52.154673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.154551    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w" pod "d2f9b794-4f22-4e6d-a851-b5d5dda79e38" (UID: "d2f9b794-4f22-4e6d-a851-b5d5dda79e38"). InnerVolumeSpecName "kube-api-access-h7h7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:52.248468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.248365    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:52.248468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.248410    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvglj\" (UniqueName: \"kubernetes.io/projected/b3e16960-7229-4ea9-a864-71f4449345cc-kube-api-access-kvglj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:52.248468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.248427    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7h7w\" (UniqueName: \"kubernetes.io/projected/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-kube-api-access-h7h7w\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:52.248468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.248441    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:52.248468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.248455    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f9b794-4f22-4e6d-a851-b5d5dda79e38-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:52.248468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.248469    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3e16960-7229-4ea9-a864-71f4449345cc-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:52.386852 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:52.386804    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582e7e73_ef9d_43ac_9717_f277deaf5ca5.slice/crio-df461f02a3116824bcadc1448ba3c7f2a7cc6be22e337218efa60a3697938941 WatchSource:0}: Error finding container df461f02a3116824bcadc1448ba3c7f2a7cc6be22e337218efa60a3697938941: Status 404 returned error can't find the container with id df461f02a3116824bcadc1448ba3c7f2a7cc6be22e337218efa60a3697938941
Nov 02 18:00:52.387292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.387268    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"]
Nov 02 18:00:52.703822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.703777    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47" event={"ID":"582e7e73-ef9d-43ac-9717-f277deaf5ca5","Type":"ContainerStarted","Data":"df461f02a3116824bcadc1448ba3c7f2a7cc6be22e337218efa60a3697938941"}
Nov 02 18:00:52.705290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.705270    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"
Nov 02 18:00:52.705425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.705299    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9" event={"ID":"b3e16960-7229-4ea9-a864-71f4449345cc","Type":"ContainerDied","Data":"d3b1026cff94903b33bd48508249cb3cfed485a9d0234830daecb6e97fc7909c"}
Nov 02 18:00:52.705425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.705346    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b1026cff94903b33bd48508249cb3cfed485a9d0234830daecb6e97fc7909c"
Nov 02 18:00:52.707035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.707012    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"
Nov 02 18:00:52.707035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.707023    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn" event={"ID":"d2f9b794-4f22-4e6d-a851-b5d5dda79e38","Type":"ContainerDied","Data":"5714327053f3531a92b94d07b9867f3f81732be2c5c6b42316324a17aee35b47"}
Nov 02 18:00:52.707223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:52.707063    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5714327053f3531a92b94d07b9867f3f81732be2c5c6b42316324a17aee35b47"
Nov 02 18:00:53.722709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:53.722677    5042 generic.go:356] "Generic (PLEG): container finished" podID="582e7e73-ef9d-43ac-9717-f277deaf5ca5" containerID="d98c8107171090e580cfd9b05a5584b5a2d1d9b410178218afa70d13a7111fd1" exitCode=0
Nov 02 18:00:53.723098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:53.722717    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47" event={"ID":"582e7e73-ef9d-43ac-9717-f277deaf5ca5","Type":"ContainerDied","Data":"d98c8107171090e580cfd9b05a5584b5a2d1d9b410178218afa70d13a7111fd1"}
Nov 02 18:00:55.176864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.176139    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:55.301446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.301398    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca\") pod \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") "
Nov 02 18:00:55.301675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.301480    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk\" (UniqueName: \"kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk\") pod \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") "
Nov 02 18:00:55.301675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.301584    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca\") pod \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\" (UID: \"582e7e73-ef9d-43ac-9717-f277deaf5ca5\") "
Nov 02 18:00:55.302973 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.302940    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca" pod "582e7e73-ef9d-43ac-9717-f277deaf5ca5" (UID: "582e7e73-ef9d-43ac-9717-f277deaf5ca5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:55.304455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.304426    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca" pod "582e7e73-ef9d-43ac-9717-f277deaf5ca5" (UID: "582e7e73-ef9d-43ac-9717-f277deaf5ca5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:55.304583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.304460    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk" pod "582e7e73-ef9d-43ac-9717-f277deaf5ca5" (UID: "582e7e73-ef9d-43ac-9717-f277deaf5ca5"). InnerVolumeSpecName "kube-api-access-7dqtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:55.404013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.403912    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:55.404013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.403947    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dqtk\" (UniqueName: \"kubernetes.io/projected/582e7e73-ef9d-43ac-9717-f277deaf5ca5-kube-api-access-7dqtk\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:55.404013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.403957    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582e7e73-ef9d-43ac-9717-f277deaf5ca5-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:00:55.793441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.793404    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"
Nov 02 18:00:55.793441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.793427    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47" event={"ID":"582e7e73-ef9d-43ac-9717-f277deaf5ca5","Type":"ContainerDied","Data":"df461f02a3116824bcadc1448ba3c7f2a7cc6be22e337218efa60a3697938941"}
Nov 02 18:00:55.793737 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:55.793491    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df461f02a3116824bcadc1448ba3c7f2a7cc6be22e337218efa60a3697938941"
Nov 02 18:00:57.218121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.218082    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"]
Nov 02 18:00:57.219179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219158    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b3e16960-7229-4ea9-a864-71f4449345cc" containerName="revoke-keys"
Nov 02 18:00:57.219302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219180    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3e16960-7229-4ea9-a864-71f4449345cc" containerName="revoke-keys"
Nov 02 18:00:57.219302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219223    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d2f9b794-4f22-4e6d-a851-b5d5dda79e38" containerName="revoke-keys"
Nov 02 18:00:57.219302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219230    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f9b794-4f22-4e6d-a851-b5d5dda79e38" containerName="revoke-keys"
Nov 02 18:00:57.219302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219256    5042 cpu_manager.go:414] "containerMap: removing container" podUID="582e7e73-ef9d-43ac-9717-f277deaf5ca5" containerName="revoke-keys"
Nov 02 18:00:57.219302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219264    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582e7e73-ef9d-43ac-9717-f277deaf5ca5" containerName="revoke-keys"
Nov 02 18:00:57.219577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219427    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3e16960-7229-4ea9-a864-71f4449345cc" containerName="revoke-keys"
Nov 02 18:00:57.219577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219442    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f9b794-4f22-4e6d-a851-b5d5dda79e38" containerName="revoke-keys"
Nov 02 18:00:57.219577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.219453    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="582e7e73-ef9d-43ac-9717-f277deaf5ca5" containerName="revoke-keys"
Nov 02 18:00:57.223212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.223188    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.237030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.236998    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-42d786-openshift-default-dockercfg-dkzz6\""
Nov 02 18:00:57.242399 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.242363    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"]
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433554    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433604    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7bj\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433652    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433731    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433816    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433863    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.433979    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.434095    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.448861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.434136    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601223    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601295    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601532    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601559    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7bj\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601603    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601681    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601778    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.601806    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.602334    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.603080    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.603682    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.605913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.605797    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.606694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.606366    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.606694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.606615    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.663717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.663639    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.706946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.700616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7bj\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.706946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.701145    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.752872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.750951    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:57.869460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:57.869357    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:58.212195 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:58.212165    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd705464_df0f_4d2f_bb88_3f9ef9057ad6.slice/crio-d8c4337c2eff5282c51d2f50d28754b54ddc03476ace1a61c15fcffa82ff224c WatchSource:0}: Error finding container d8c4337c2eff5282c51d2f50d28754b54ddc03476ace1a61c15fcffa82ff224c: Status 404 returned error can't find the container with id d8c4337c2eff5282c51d2f50d28754b54ddc03476ace1a61c15fcffa82ff224c
Nov 02 18:00:58.215457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.215034    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"]
Nov 02 18:00:58.215457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.215243    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:58.215457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.215319    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:58.215457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.215349    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:58.243005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.242976    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"]
Nov 02 18:00:58.249832 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.249804    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.254043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.254011    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:00:58.254171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.254085    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:00:58.254908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.254833    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:00:58.255310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.255288    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:00:58.255953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.255928    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"]
Nov 02 18:00:58.354484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.354446    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49r82\" (UniqueName: \"kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.354691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.354649    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.354856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.354812    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.372780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.358940    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"]
Nov 02 18:00:58.392142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.392104    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"]
Nov 02 18:00:58.392467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.392446    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.456874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.456816    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.457051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.456969    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.457203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.457187    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49r82\" (UniqueName: \"kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.457587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.457567    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.459634 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.459575    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.467043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.467018    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49r82\" (UniqueName: \"kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82\") pod \"maas-api-revoke-sub-8d1da6fe-m4z6r\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.559065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.559021    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldbc\" (UniqueName: \"kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.559250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.559078    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.559337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.559306    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.590889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.590824    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:00:58.663250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.662387    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldbc\" (UniqueName: \"kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.663250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.662451    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.663250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.662575    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.664555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.664496    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.666343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.666317    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.679461 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.679407    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldbc\" (UniqueName: \"kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc\") pod \"maas-api-revoke-sub-d6252674-hd4qr\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.750943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.750823    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:00:58.852734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.852699    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" event={"ID":"dd705464-df0f-4d2f-bb88-3f9ef9057ad6","Type":"ContainerStarted","Data":"c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"}
Nov 02 18:00:58.852991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.852971    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" event={"ID":"dd705464-df0f-4d2f-bb88-3f9ef9057ad6","Type":"ContainerStarted","Data":"d8c4337c2eff5282c51d2f50d28754b54ddc03476ace1a61c15fcffa82ff224c"}
Nov 02 18:00:58.870140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.870108    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:58.886365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.886304    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podStartSLOduration=1.886282342 podStartE2EDuration="1.886282342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:58.880061989 +0000 UTC m=+1671.459471375" watchObservedRunningTime="2026-11-02 18:00:58.886282342 +0000 UTC m=+1671.465691729"
Nov 02 18:00:58.896874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:58.895916    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"]
Nov 02 18:00:59.146417 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:00:59.146379    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4803ff45_a5a9_4093_b1c9_a8244bcac5bb.slice/crio-1b0bc3534f745fcfd4d4d0a949b24b703318c962fb6de08ecc5660584535dd9c WatchSource:0}: Error finding container 1b0bc3534f745fcfd4d4d0a949b24b703318c962fb6de08ecc5660584535dd9c: Status 404 returned error can't find the container with id 1b0bc3534f745fcfd4d4d0a949b24b703318c962fb6de08ecc5660584535dd9c
Nov 02 18:00:59.149547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.149489    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"]
Nov 02 18:00:59.864206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.864166    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr" event={"ID":"4803ff45-a5a9-4093-b1c9-a8244bcac5bb","Type":"ContainerStarted","Data":"0e097a678aee6a309ce38e00a4dc07c551d0e1e08e58f6db8d3c9050df36106e"}
Nov 02 18:00:59.864681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.864227    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr" event={"ID":"4803ff45-a5a9-4093-b1c9-a8244bcac5bb","Type":"ContainerStarted","Data":"1b0bc3534f745fcfd4d4d0a949b24b703318c962fb6de08ecc5660584535dd9c"}
Nov 02 18:00:59.866202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.866173    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r" event={"ID":"5b1eb281-4071-4e0b-8b2f-43a7e59dee07","Type":"ContainerStarted","Data":"191a7f66245422bc42cc8c83d983859752a1b924a81eb491f6c3d13077a58ed6"}
Nov 02 18:00:59.866342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.866222    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r" event={"ID":"5b1eb281-4071-4e0b-8b2f-43a7e59dee07","Type":"ContainerStarted","Data":"cc14391bfc37f1be73a3efca857a1a1f8dff7be1a0f77d2f7a1fb7581851678d"}
Nov 02 18:00:59.878222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.878181    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:00:59.883854 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.883797    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr" podStartSLOduration=1.8837826180000001 podStartE2EDuration="1.883782618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:59.87972364 +0000 UTC m=+1672.459133023" watchObservedRunningTime="2026-11-02 18:00:59.883782618 +0000 UTC m=+1672.463192004"
Nov 02 18:00:59.896461 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:00:59.896411    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r" podStartSLOduration=1.896393471 podStartE2EDuration="1.896393471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:59.895665207 +0000 UTC m=+1672.475074595" watchObservedRunningTime="2026-11-02 18:00:59.896393471 +0000 UTC m=+1672.475802858"
Nov 02 18:01:00.879909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:00.879871    5042 generic.go:356] "Generic (PLEG): container finished" podID="4803ff45-a5a9-4093-b1c9-a8244bcac5bb" containerID="0e097a678aee6a309ce38e00a4dc07c551d0e1e08e58f6db8d3c9050df36106e" exitCode=0
Nov 02 18:01:00.880418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:00.879955    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr" event={"ID":"4803ff45-a5a9-4093-b1c9-a8244bcac5bb","Type":"ContainerDied","Data":"0e097a678aee6a309ce38e00a4dc07c551d0e1e08e58f6db8d3c9050df36106e"}
Nov 02 18:01:00.882012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:00.881987    5042 generic.go:356] "Generic (PLEG): container finished" podID="5b1eb281-4071-4e0b-8b2f-43a7e59dee07" containerID="191a7f66245422bc42cc8c83d983859752a1b924a81eb491f6c3d13077a58ed6" exitCode=0
Nov 02 18:01:00.882170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:00.882067    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r" event={"ID":"5b1eb281-4071-4e0b-8b2f-43a7e59dee07","Type":"ContainerDied","Data":"191a7f66245422bc42cc8c83d983859752a1b924a81eb491f6c3d13077a58ed6"}
Nov 02 18:01:00.882277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:00.882252    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:01:00.883578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:00.883559    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:01:02.039938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.037729    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"]
Nov 02 18:01:02.075682 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.075613    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.080560 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.079054    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"]
Nov 02 18:01:02.162566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.162525    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.162886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.162825    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lftw\" (UniqueName: \"kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.163058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.163039    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.177124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.177098    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:01:02.180022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.179996    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:01:02.264359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.264102    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca\") pod \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") "
Nov 02 18:01:02.264359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.264256    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc\" (UniqueName: \"kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc\") pod \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") "
Nov 02 18:01:02.264993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.264963    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca\") pod \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") "
Nov 02 18:01:02.265345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.265038    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca\") pod \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\" (UID: \"4803ff45-a5a9-4093-b1c9-a8244bcac5bb\") "
Nov 02 18:01:02.265345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.265108    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82\" (UniqueName: \"kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82\") pod \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") "
Nov 02 18:01:02.265345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.265161    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca\") pod \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\" (UID: \"5b1eb281-4071-4e0b-8b2f-43a7e59dee07\") "
Nov 02 18:01:02.265612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.265583    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca" pod "4803ff45-a5a9-4093-b1c9-a8244bcac5bb" (UID: "4803ff45-a5a9-4093-b1c9-a8244bcac5bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:02.266438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.266407    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.266808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.266783    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lftw\" (UniqueName: \"kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.267090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.267066    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.269510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.267257    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:02.269510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.267546    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca" pod "5b1eb281-4071-4e0b-8b2f-43a7e59dee07" (UID: "5b1eb281-4071-4e0b-8b2f-43a7e59dee07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:02.269510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.267821    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca" pod "5b1eb281-4071-4e0b-8b2f-43a7e59dee07" (UID: "5b1eb281-4071-4e0b-8b2f-43a7e59dee07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:02.269510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.268629    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.271448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.271421    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc" pod "4803ff45-a5a9-4093-b1c9-a8244bcac5bb" (UID: "4803ff45-a5a9-4093-b1c9-a8244bcac5bb"). InnerVolumeSpecName "kube-api-access-cldbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:02.293111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.272987    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.293111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.276702    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lftw\" (UniqueName: \"kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw\") pod \"maas-api-revoke-sub-a5cf42b8-m59qd\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.293111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.276934    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82" pod "5b1eb281-4071-4e0b-8b2f-43a7e59dee07" (UID: "5b1eb281-4071-4e0b-8b2f-43a7e59dee07"). InnerVolumeSpecName "kube-api-access-49r82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:02.293111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.280231    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca" pod "4803ff45-a5a9-4093-b1c9-a8244bcac5bb" (UID: "4803ff45-a5a9-4093-b1c9-a8244bcac5bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:02.346142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.346109    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"]
Nov 02 18:01:02.347091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.347075    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5b1eb281-4071-4e0b-8b2f-43a7e59dee07" containerName="revoke-keys"
Nov 02 18:01:02.347145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.347093    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1eb281-4071-4e0b-8b2f-43a7e59dee07" containerName="revoke-keys"
Nov 02 18:01:02.347145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.347123    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4803ff45-a5a9-4093-b1c9-a8244bcac5bb" containerName="revoke-keys"
Nov 02 18:01:02.347145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.347128    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4803ff45-a5a9-4093-b1c9-a8244bcac5bb" containerName="revoke-keys"
Nov 02 18:01:02.347241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.347230    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1eb281-4071-4e0b-8b2f-43a7e59dee07" containerName="revoke-keys"
Nov 02 18:01:02.347278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.347245    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4803ff45-a5a9-4093-b1c9-a8244bcac5bb" containerName="revoke-keys"
Nov 02 18:01:02.351996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.351978    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.356126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.356097    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-42d786-dockercfg-5dr7s\""
Nov 02 18:01:02.356360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.356337    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-42d786\""
Nov 02 18:01:02.362875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.362821    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"]
Nov 02 18:01:02.370268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.370128    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:02.370437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.370281    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49r82\" (UniqueName: \"kubernetes.io/projected/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-kube-api-access-49r82\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:02.370437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.370301    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:02.370437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.370314    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b1eb281-4071-4e0b-8b2f-43a7e59dee07-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:02.370437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.370332    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cldbc\" (UniqueName: \"kubernetes.io/projected/4803ff45-a5a9-4093-b1c9-a8244bcac5bb-kube-api-access-cldbc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:02.394617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.394581    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"]
Nov 02 18:01:02.408250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.405779    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.419764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.419000    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"]
Nov 02 18:01:02.472962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.472367    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fml\" (UniqueName: \"kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml\") pod \"payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.472962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.472619    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstxk\" (UniqueName: \"kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk\") pod \"payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.472962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.472682    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:02.472962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.472883    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.473382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.472981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.644239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.617603    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.644239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.617694    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.644239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.640995    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fml\" (UniqueName: \"kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml\") pod \"payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.644239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.641056    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstxk\" (UniqueName: \"kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk\") pod \"payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.644239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.642962    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.665329 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.664697    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.685637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.676080    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstxk\" (UniqueName: \"kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk\") pod \"payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.685637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.685567    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fml\" (UniqueName: \"kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml\") pod \"payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.761867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.744738    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:02.971737 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.971440    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:02.992112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.992088    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"
Nov 02 18:01:02.992885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.992853    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr" event={"ID":"4803ff45-a5a9-4093-b1c9-a8244bcac5bb","Type":"ContainerDied","Data":"1b0bc3534f745fcfd4d4d0a949b24b703318c962fb6de08ecc5660584535dd9c"}
Nov 02 18:01:02.993103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.993087    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0bc3534f745fcfd4d4d0a949b24b703318c962fb6de08ecc5660584535dd9c"
Nov 02 18:01:02.996006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.995988    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"
Nov 02 18:01:02.997216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.997065    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r" event={"ID":"5b1eb281-4071-4e0b-8b2f-43a7e59dee07","Type":"ContainerDied","Data":"cc14391bfc37f1be73a3efca857a1a1f8dff7be1a0f77d2f7a1fb7581851678d"}
Nov 02 18:01:02.997216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:02.997122    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc14391bfc37f1be73a3efca857a1a1f8dff7be1a0f77d2f7a1fb7581851678d"
Nov 02 18:01:03.029698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:03.029656    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"]
Nov 02 18:01:03.035564 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:03.035532    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0545acec-6hxkb"]
Nov 02 18:01:03.247934 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:01:03.247902    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d626b2f_9de2_47b7_be14_0e0ee402715a.slice/crio-1eb22ec1cfcb606c6ff2d27a4a5054fe0d2027562f83769770d00afd27a45808 WatchSource:0}: Error finding container 1eb22ec1cfcb606c6ff2d27a4a5054fe0d2027562f83769770d00afd27a45808: Status 404 returned error can't find the container with id 1eb22ec1cfcb606c6ff2d27a4a5054fe0d2027562f83769770d00afd27a45808
Nov 02 18:01:03.248722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:03.248701    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"]
Nov 02 18:01:03.312001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:03.311969    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"]
Nov 02 18:01:03.315405 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:01:03.315367    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e8c557_0b90_4592_b388_fc08b2fcc5b5.slice/crio-65b9066b9e0fcc3c7731c9ec274c6f674bc7c17ff786a7bf23e45b9b605e47c7 WatchSource:0}: Error finding container 65b9066b9e0fcc3c7731c9ec274c6f674bc7c17ff786a7bf23e45b9b605e47c7: Status 404 returned error can't find the container with id 65b9066b9e0fcc3c7731c9ec274c6f674bc7c17ff786a7bf23e45b9b605e47c7
Nov 02 18:01:03.347406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:03.347380    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"]
Nov 02 18:01:03.553584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:03.553544    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2c1132-690f-4153-9184-1feae3dea181" path="/var/lib/kubelet/pods/fc2c1132-690f-4153-9184-1feae3dea181/volumes"
Nov 02 18:01:04.006607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.006564    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" event={"ID":"01716d13-d310-4be8-8c2a-f25015dc4e83","Type":"ContainerStarted","Data":"efcbbddda0ca2a279f6d7c6f23e0d3d145334a146c12c8bec2f62bf423014e5b"}
Nov 02 18:01:04.006962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.006940    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:04.007104 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.006985    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" event={"ID":"01716d13-d310-4be8-8c2a-f25015dc4e83","Type":"ContainerStarted","Data":"f9c01ea50cf5a73a735c0a50f86ec2d67f125bccb86f4767c1da9f0bc00fe684"}
Nov 02 18:01:04.008802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.008757    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd" event={"ID":"5d626b2f-9de2-47b7-be14-0e0ee402715a","Type":"ContainerStarted","Data":"8dbbffea7337a784f0a72bb23a370126089f5eb04a0b10c7c2b46c5967e6ca2e"}
Nov 02 18:01:04.008925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.008818    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd" event={"ID":"5d626b2f-9de2-47b7-be14-0e0ee402715a","Type":"ContainerStarted","Data":"1eb22ec1cfcb606c6ff2d27a4a5054fe0d2027562f83769770d00afd27a45808"}
Nov 02 18:01:04.010969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.010941    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" event={"ID":"54e8c557-0b90-4592-b388-fc08b2fcc5b5","Type":"ContainerStarted","Data":"af0a9718ef882e647babae4b35065357c8e7411679390981fd7e45ad8ea2524a"}
Nov 02 18:01:04.011069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.010990    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" event={"ID":"54e8c557-0b90-4592-b388-fc08b2fcc5b5","Type":"ContainerStarted","Data":"65b9066b9e0fcc3c7731c9ec274c6f674bc7c17ff786a7bf23e45b9b605e47c7"}
Nov 02 18:01:04.011069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.011043    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:04.048597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.048542    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" podStartSLOduration=2.048528078 podStartE2EDuration="2.048528078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:04.046029187 +0000 UTC m=+1676.625438574" watchObservedRunningTime="2026-11-02 18:01:04.048528078 +0000 UTC m=+1676.627937464"
Nov 02 18:01:04.080879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.080809    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" podStartSLOduration=2.080796165 podStartE2EDuration="2.080796165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:04.07110765 +0000 UTC m=+1676.650517037" watchObservedRunningTime="2026-11-02 18:01:04.080796165 +0000 UTC m=+1676.660205587"
Nov 02 18:01:04.107123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:04.107074    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd" podStartSLOduration=2.107060549 podStartE2EDuration="2.107060549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:04.106595198 +0000 UTC m=+1676.686004585" watchObservedRunningTime="2026-11-02 18:01:04.107060549 +0000 UTC m=+1676.686469934"
Nov 02 18:01:05.025778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.025742    5042 generic.go:356] "Generic (PLEG): container finished" podID="5d626b2f-9de2-47b7-be14-0e0ee402715a" containerID="8dbbffea7337a784f0a72bb23a370126089f5eb04a0b10c7c2b46c5967e6ca2e" exitCode=0
Nov 02 18:01:05.026301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.025821    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd" event={"ID":"5d626b2f-9de2-47b7-be14-0e0ee402715a","Type":"ContainerDied","Data":"8dbbffea7337a784f0a72bb23a370126089f5eb04a0b10c7c2b46c5967e6ca2e"}
Nov 02 18:01:05.630775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.630558    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"]
Nov 02 18:01:05.636956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.636923    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.652333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.652305    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5ec436-openshift-default-dockercfg-wqbgt\""
Nov 02 18:01:05.657337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.656935    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"]
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730390    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730490    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730529    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730564    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8lf\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730634    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730787    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.730870    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.731012    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.774873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.731079    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.907924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.837655    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.907924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.837718    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.907924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.837753    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8lf\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.907924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.849524    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.907924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.849789    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.908984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.908953    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.909138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.837828    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.950947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.913055    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.951286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.951257    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.951528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.951512    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.951666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.951652    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.951908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.951892    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.952872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.952825    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.953048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.918537    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:05.953140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.953071    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8lf\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:06.013864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:05.982607    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:06.014936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.014905    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:06.015767 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.015743    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:06.263876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.263252    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:06.519308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.478149    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:06.642301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.642268    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca\") pod \"5d626b2f-9de2-47b7-be14-0e0ee402715a\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") "
Nov 02 18:01:06.642462 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.642396    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw\" (UniqueName: \"kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw\") pod \"5d626b2f-9de2-47b7-be14-0e0ee402715a\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") "
Nov 02 18:01:06.642508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.642480    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca\") pod \"5d626b2f-9de2-47b7-be14-0e0ee402715a\" (UID: \"5d626b2f-9de2-47b7-be14-0e0ee402715a\") "
Nov 02 18:01:06.642642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.642619    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca" pod "5d626b2f-9de2-47b7-be14-0e0ee402715a" (UID: "5d626b2f-9de2-47b7-be14-0e0ee402715a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:06.643272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.643251    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:06.644623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.644593    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca" pod "5d626b2f-9de2-47b7-be14-0e0ee402715a" (UID: "5d626b2f-9de2-47b7-be14-0e0ee402715a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:06.644749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.644733    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw" pod "5d626b2f-9de2-47b7-be14-0e0ee402715a" (UID: "5d626b2f-9de2-47b7-be14-0e0ee402715a"). InnerVolumeSpecName "kube-api-access-5lftw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:06.745730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.745702    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d626b2f-9de2-47b7-be14-0e0ee402715a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:06.745789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.745742    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lftw\" (UniqueName: \"kubernetes.io/projected/5d626b2f-9de2-47b7-be14-0e0ee402715a-kube-api-access-5lftw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:06.748251 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:01:06.748220    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74beb22_3816_4661_aa11_2c034967e021.slice/crio-6247b8d85d68b1199c676deeaff2c50118b08c3daf1815d9e41f4aa834fdcd1e WatchSource:0}: Error finding container 6247b8d85d68b1199c676deeaff2c50118b08c3daf1815d9e41f4aa834fdcd1e: Status 404 returned error can't find the container with id 6247b8d85d68b1199c676deeaff2c50118b08c3daf1815d9e41f4aa834fdcd1e
Nov 02 18:01:06.748519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.748496    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"]
Nov 02 18:01:06.751283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.751250    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:06.751364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.751321    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:06.751364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:06.751353    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:07.113218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.113121    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd" event={"ID":"5d626b2f-9de2-47b7-be14-0e0ee402715a","Type":"ContainerDied","Data":"1eb22ec1cfcb606c6ff2d27a4a5054fe0d2027562f83769770d00afd27a45808"}
Nov 02 18:01:07.113218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.113205    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eb22ec1cfcb606c6ff2d27a4a5054fe0d2027562f83769770d00afd27a45808"
Nov 02 18:01:07.113473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.113266    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"
Nov 02 18:01:07.121557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.121526    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" event={"ID":"c74beb22-3816-4661-aa11-2c034967e021","Type":"ContainerStarted","Data":"830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"}
Nov 02 18:01:07.121740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.121586    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" event={"ID":"c74beb22-3816-4661-aa11-2c034967e021","Type":"ContainerStarted","Data":"6247b8d85d68b1199c676deeaff2c50118b08c3daf1815d9e41f4aa834fdcd1e"}
Nov 02 18:01:07.171875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.165025    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podStartSLOduration=2.165002079 podStartE2EDuration="2.165002079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:07.147329838 +0000 UTC m=+1679.726739226" watchObservedRunningTime="2026-11-02 18:01:07.165002079 +0000 UTC m=+1679.744411466"
Nov 02 18:01:07.276863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.276377    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:07.301863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.292983    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:01:07.301863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:07.293049    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:01:08.467865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:08.460167    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:09.146050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:09.146019    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:09.147223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:09.147193    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:01:10.476861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:10.455855    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"]
Nov 02 18:01:11.056250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.056219    5042 scope.go:122] "RemoveContainer" containerID="268b59ed6069f7089d12a70e5a0dd1d00844dcf4035b4e81c75d3ab1545492c7"
Nov 02 18:01:11.080881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.080829    5042 scope.go:122] "RemoveContainer" containerID="66542c6df2dbf14a430187a8008ee1a4e0d0b4411cdbe5c56c25d20e0e4ae726"
Nov 02 18:01:11.202334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.198616    5042 scope.go:122] "RemoveContainer" containerID="701c0737e9ddba1d5f8e28ddb08359134ead0694f1390d85c7f1e428f402b9ab"
Nov 02 18:01:11.345877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.342129    5042 scope.go:122] "RemoveContainer" containerID="6e2baee3302593c9bd285c4a487da3a6b9a3e30db8b24a80ef52b6896c1b9d15"
Nov 02 18:01:11.435566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.435447    5042 scope.go:122] "RemoveContainer" containerID="3f1f05d38425fbd72972ec933590e9aefdbedf3cc51b8df1a2d93897eaba32db"
Nov 02 18:01:11.456348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.456324    5042 scope.go:122] "RemoveContainer" containerID="cfae14eb9d4cfa63754ad0888d626fd46170ad445c5498b7c8231343ef2d1cb5"
Nov 02 18:01:11.469553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.469532    5042 scope.go:122] "RemoveContainer" containerID="c6c0e1e0d7b7ede5f23f331c24784f78f13e53dcc6dca988736f6b2dad4495b1"
Nov 02 18:01:11.502429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.502402    5042 scope.go:122] "RemoveContainer" containerID="ede59465a41daeeaba9d3d14ec5b4692ce152aad30498aa4558e9eacf2b81985"
Nov 02 18:01:11.591130 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:11.591031    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"]
Nov 02 18:01:12.799869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.796167    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"]
Nov 02 18:01:12.799869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.797503    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5d626b2f-9de2-47b7-be14-0e0ee402715a" containerName="revoke-keys"
Nov 02 18:01:12.799869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.797526    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d626b2f-9de2-47b7-be14-0e0ee402715a" containerName="revoke-keys"
Nov 02 18:01:12.799869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.797702    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d626b2f-9de2-47b7-be14-0e0ee402715a" containerName="revoke-keys"
Nov 02 18:01:12.847389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.847358    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:12.848811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.848789    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"]
Nov 02 18:01:12.864261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:12.863068    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4ea582-openshift-default-dockercfg-4stc9\""
Nov 02 18:01:13.432870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.431669    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.534652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.534471    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.534652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.534556    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6bg\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.534652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.534658    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.537734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.534750    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.537734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.534794    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.537734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.534948    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.537734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.536016    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.537734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.536112    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.730527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.730485    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.731551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.731528    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.732467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.732445    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.733255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.733234    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.734158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.734134    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.734338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.734317    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6bg\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.734553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.734534    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.736402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.736383    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.737174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.737156    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.737878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.735505    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.738025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.733984    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.738131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.731426    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.738221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.737086    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.738308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.733055    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.757979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.754575    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.757979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.755358    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.757979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.757015    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6bg\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:13.759534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:13.759507    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:14.051923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.046970    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:14.523543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.522389    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"]
Nov 02 18:01:14.525321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.525261    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:14.526007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.525982    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:14.527297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.526986    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:14.755140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.755054    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" event={"ID":"87c8e2ba-6618-44e0-80c0-7823fdfeba6a","Type":"ContainerStarted","Data":"d3193892c26145e1d67b700784c204cc8826561fa2977d79a46d78f815e12fad"}
Nov 02 18:01:14.755140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.755118    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" event={"ID":"87c8e2ba-6618-44e0-80c0-7823fdfeba6a","Type":"ContainerStarted","Data":"3eac565d84e90db184bcb5a0480163a49a21e6477cd3b02da5ad1aada5326e95"}
Nov 02 18:01:14.797865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:14.797483    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podStartSLOduration=2.7974601569999997 podStartE2EDuration="2.797460157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:14.796138618 +0000 UTC m=+1687.375548031" watchObservedRunningTime="2026-11-02 18:01:14.797460157 +0000 UTC m=+1687.376869543"
Nov 02 18:01:15.060922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:15.052994    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:15.060922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:15.058454    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:01:15.060922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:15.060050    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:01:15.060922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:15.060116    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:01:15.184867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:15.184054    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:01:16.053987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:16.053956    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:16.815453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:16.815419    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:16.817051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:16.817024    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:01:19.002652 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:01:19.002561    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:01:21.457395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:21.457350    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"]
Nov 02 18:01:25.150137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:25.134965    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:01:25.150137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:25.135035    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:26.232805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.230587    5042 generic.go:356] "Generic (PLEG): container finished" podID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerID="0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38" exitCode=0
Nov 02 18:01:26.232805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.231004    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" event={"ID":"c04ba0c7-e939-4825-81be-080c7b4139d9","Type":"ContainerDied","Data":"0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"}
Nov 02 18:01:26.232805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.231687    5042 scope.go:122] "RemoveContainer" containerID="0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"
Nov 02 18:01:26.242522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.242477    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:26.242739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.242582    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:26.242739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.242631    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:26.424083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.424043    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" containerID="cri-o://60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188" gracePeriod=45
Nov 02 18:01:26.642888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:26.642204    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" podUID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" containerName="main" containerID="cri-o://a9dc045c8474fbcc5ffd55e9106ab3f60fa006aa35e646ba5057e24f901ee836" gracePeriod=45
Nov 02 18:01:27.114395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.113037    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 18:01:27.213581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.213337    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs\") pod \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") "
Nov 02 18:01:27.213995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.213965    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv\" (UniqueName: \"kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv\") pod \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") "
Nov 02 18:01:27.214389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.214368    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir\") pod \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") "
Nov 02 18:01:27.214568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.214550    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache\") pod \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") "
Nov 02 18:01:27.215076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.215056    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm\") pod \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") "
Nov 02 18:01:27.215222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.215207    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home\") pod \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\" (UID: \"cd74de39-73ad-4ad7-9bac-ce575fb6219d\") "
Nov 02 18:01:27.219168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.217700    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home" pod "cd74de39-73ad-4ad7-9bac-ce575fb6219d" (UID: "cd74de39-73ad-4ad7-9bac-ce575fb6219d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.220864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.220816    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache" pod "cd74de39-73ad-4ad7-9bac-ce575fb6219d" (UID: "cd74de39-73ad-4ad7-9bac-ce575fb6219d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.221350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.221328    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir" pod "cd74de39-73ad-4ad7-9bac-ce575fb6219d" (UID: "cd74de39-73ad-4ad7-9bac-ce575fb6219d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.252433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.244650    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm" pod "cd74de39-73ad-4ad7-9bac-ce575fb6219d" (UID: "cd74de39-73ad-4ad7-9bac-ce575fb6219d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.278233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.276059    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv" pod "cd74de39-73ad-4ad7-9bac-ce575fb6219d" (UID: "cd74de39-73ad-4ad7-9bac-ce575fb6219d"). InnerVolumeSpecName "kube-api-access-vcnjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:27.282880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.279991    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs" pod "cd74de39-73ad-4ad7-9bac-ce575fb6219d" (UID: "cd74de39-73ad-4ad7-9bac-ce575fb6219d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:27.339942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.321787    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.367809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.367775    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.367809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.367809    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.368097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.367824    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnjv\" (UniqueName: \"kubernetes.io/projected/cd74de39-73ad-4ad7-9bac-ce575fb6219d-kube-api-access-vcnjv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.368097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.367856    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.368097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.367872    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74de39-73ad-4ad7-9bac-ce575fb6219d-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.375149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.373152    5042 generic.go:356] "Generic (PLEG): container finished" podID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" containerID="a9dc045c8474fbcc5ffd55e9106ab3f60fa006aa35e646ba5057e24f901ee836" exitCode=0
Nov 02 18:01:27.375149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.373316    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" event={"ID":"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37","Type":"ContainerDied","Data":"a9dc045c8474fbcc5ffd55e9106ab3f60fa006aa35e646ba5057e24f901ee836"}
Nov 02 18:01:27.383962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.378935    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 18:01:27.383962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.379921    5042 generic.go:356] "Generic (PLEG): container finished" podID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerID="60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188" exitCode=0
Nov 02 18:01:27.383962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.379979    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"
Nov 02 18:01:27.383962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.380023    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" event={"ID":"cd74de39-73ad-4ad7-9bac-ce575fb6219d","Type":"ContainerDied","Data":"60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188"}
Nov 02 18:01:27.383962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.380056    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" event={"ID":"cd74de39-73ad-4ad7-9bac-ce575fb6219d","Type":"ContainerDied","Data":"d3899e623afffc69405d66dd93fb0c4f94f49b27116e3bafe4028a5ce3a86870"}
Nov 02 18:01:27.383962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.380077    5042 scope.go:122] "RemoveContainer" containerID="60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188"
Nov 02 18:01:27.386230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.386185    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" event={"ID":"c04ba0c7-e939-4825-81be-080c7b4139d9","Type":"ContainerStarted","Data":"9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161"}
Nov 02 18:01:27.413084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.406088    5042 scope.go:122] "RemoveContainer" containerID="60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188"
Nov 02 18:01:27.413084 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:01:27.408420    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188\": container with ID starting with 60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188 not found: ID does not exist" containerID="60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188"
Nov 02 18:01:27.413084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.408462    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188"} err="failed to get container status \"60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188\": rpc error: code = NotFound desc = could not find container \"60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188\": container with ID starting with 60c34648c8a6b26b7f56e34cb57cacdac1a27319f43355bfb107217684a3a188 not found: ID does not exist"
Nov 02 18:01:27.436947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.431434    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"]
Nov 02 18:01:27.443865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.442203    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l"]
Nov 02 18:01:27.468635 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.468537    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache\") pod \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") "
Nov 02 18:01:27.468635 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.468608    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4\" (UniqueName: \"kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4\") pod \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") "
Nov 02 18:01:27.468952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.468652    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm\") pod \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") "
Nov 02 18:01:27.468952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.468944    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs\") pod \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") "
Nov 02 18:01:27.469080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.468986    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home\") pod \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") "
Nov 02 18:01:27.469080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.468942    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache" pod "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" (UID: "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.469080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.469007    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir\") pod \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\" (UID: \"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37\") "
Nov 02 18:01:27.469278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.469219    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home" pod "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" (UID: "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.469454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.469284    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir" pod "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" (UID: "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.470026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.469999    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.470134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.470036    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.470134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.470058    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.471183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.471160    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm" pod "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" (UID: "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:27.471306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.471288    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs" pod "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" (UID: "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:27.471372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.471334    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4" pod "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" (UID: "4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37"). InnerVolumeSpecName "kube-api-access-c6kx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:27.549389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.549355    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" path="/var/lib/kubelet/pods/cd74de39-73ad-4ad7-9bac-ce575fb6219d/volumes"
Nov 02 18:01:27.577870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.572389    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6kx4\" (UniqueName: \"kubernetes.io/projected/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-kube-api-access-c6kx4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.577870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.572429    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.577870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.572443    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:27.769608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:27.769491    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-86f5f0/auth-test-model-86f5f0-kserve-7f4c4b597-k2j8l" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.196:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:28.331915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.331881    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:01:28.332811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.332783    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:28.332950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.332783    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" podUID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" containerName="main" probeResult="failure" output="Get \"https://10.132.0.207:8000/ready\": context deadline exceeded"
Nov 02 18:01:28.332950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.332850    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:01:28.415356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.415326    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"
Nov 02 18:01:28.415904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.415625    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc" event={"ID":"4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37","Type":"ContainerDied","Data":"37987a39c2941bec6622e1e2e295c2f80426b47aa03bce8545d648a7ad9a9af7"}
Nov 02 18:01:28.415904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.415674    5042 scope.go:122] "RemoveContainer" containerID="a9dc045c8474fbcc5ffd55e9106ab3f60fa006aa35e646ba5057e24f901ee836"
Nov 02 18:01:28.451898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.448367    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"]
Nov 02 18:01:28.453952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.453923    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-7771bf/auth-test-model-7771bf-kserve-db466b9b5-7kzzc"]
Nov 02 18:01:28.626802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.626711    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"]
Nov 02 18:01:28.627571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.627551    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main"
Nov 02 18:01:28.627571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.627571    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main"
Nov 02 18:01:28.627746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.627632    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" containerName="main"
Nov 02 18:01:28.627746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.627638    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" containerName="main"
Nov 02 18:01:28.627982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.627965    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" containerName="main"
Nov 02 18:01:28.628052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.627987    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd74de39-73ad-4ad7-9bac-ce575fb6219d" containerName="main"
Nov 02 18:01:28.633350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.633256    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:28.638356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.637329    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a8e574\""
Nov 02 18:01:28.638356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.637830    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a8e574-dockercfg-dlm2d\""
Nov 02 18:01:28.641641 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.641615    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"]
Nov 02 18:01:28.733867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.731543    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:28.733867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.731619    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nk8\" (UniqueName: \"kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8\") pod \"payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:28.877940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.833286    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:28.881872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.878238    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nk8\" (UniqueName: \"kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8\") pod \"payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:28.889877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:28.836408    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:29.179200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:29.179091    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nk8\" (UniqueName: \"kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8\") pod \"payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:29.255281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:29.255244    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:29.333028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:29.332993    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:29.333535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:29.333044    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:01:29.558863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:29.557816    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37" path="/var/lib/kubelet/pods/4fb03c20-d3d4-44a6-a7e4-dfe3139e2b37/volumes"
Nov 02 18:01:29.806276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:29.806240    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"]
Nov 02 18:01:29.809150 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:01:29.809072    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b99141_2c4b_4e25_9d94_c8540619fe66.slice/crio-ee20fa05e711bafd05081a91a337c16686fc7d4111b80884b049b19d2ef33157 WatchSource:0}: Error finding container ee20fa05e711bafd05081a91a337c16686fc7d4111b80884b049b19d2ef33157: Status 404 returned error can't find the container with id ee20fa05e711bafd05081a91a337c16686fc7d4111b80884b049b19d2ef33157
Nov 02 18:01:30.336238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:30.332959    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:30.336238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:30.333025    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:01:30.543658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:30.543620    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" event={"ID":"41b99141-2c4b-4e25-9d94-c8540619fe66","Type":"ContainerStarted","Data":"0a15d77c590adada9a8b11739cb8e7c97a65b3307c00b7eb4d88b02bbf006610"}
Nov 02 18:01:30.544015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:30.543683    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" event={"ID":"41b99141-2c4b-4e25-9d94-c8540619fe66","Type":"ContainerStarted","Data":"ee20fa05e711bafd05081a91a337c16686fc7d4111b80884b049b19d2ef33157"}
Nov 02 18:01:30.544015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:30.543900    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:30.565911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:30.565850    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" podStartSLOduration=2.56582155 podStartE2EDuration="2.56582155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:30.561093493 +0000 UTC m=+1703.140502879" watchObservedRunningTime="2026-11-02 18:01:30.56582155 +0000 UTC m=+1703.145230937"
Nov 02 18:01:31.333229 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:31.333190    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:31.333448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:31.333258    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:01:32.332170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:32.332131    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:32.332609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:32.332201    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:01:33.332720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.332689    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:33.333246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.333073    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Nov 02 18:01:33.368393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.368354    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"]
Nov 02 18:01:33.429766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.429737    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"]
Nov 02 18:01:33.430139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.430123    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.542278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.541991    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.542278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.542075    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.542278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.542113    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.544402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.542912    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.544402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.542966    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.544402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.543120    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.544402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.543153    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn964\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.699868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.699493    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.699868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.699561    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.699868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.699749    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.699868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.699784    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn964\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.700296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.699936    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.700296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.700000    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.700296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.700033    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.724896 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.721997    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.749983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.742724    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.783866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.777177    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.783866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.779473    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn964\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.803876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.787215    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.803876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.787873    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:33.823653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:33.823618    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tw9kl\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:34.089776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.089672    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:34.566195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.566159    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:01:34.570259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.570223    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:01:34.582544 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.582359    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:01:34.739532 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:01:34.739502    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab49f445_8557_460c_9080_8caa6e601004.slice/crio-0fad1d9c92b4a43a148bf0493024c79799ff43157080e6e4f7a273d52941cd86 WatchSource:0}: Error finding container 0fad1d9c92b4a43a148bf0493024c79799ff43157080e6e4f7a273d52941cd86: Status 404 returned error can't find the container with id 0fad1d9c92b4a43a148bf0493024c79799ff43157080e6e4f7a273d52941cd86
Nov 02 18:01:34.743410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.743376    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:34.743526 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.743442    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:34.744272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:34.744247    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"]
Nov 02 18:01:35.681864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:35.681103    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" event={"ID":"ab49f445-8557-460c-9080-8caa6e601004","Type":"ContainerStarted","Data":"13364444460ba3c8ce5f567351ef814670a11866142b71b413ceddf27b90fafa"}
Nov 02 18:01:35.681864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:35.681196    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" event={"ID":"ab49f445-8557-460c-9080-8caa6e601004","Type":"ContainerStarted","Data":"0fad1d9c92b4a43a148bf0493024c79799ff43157080e6e4f7a273d52941cd86"}
Nov 02 18:01:35.681864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:35.681472    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:35.713864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:35.708073    5042 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tw9kl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:01:35.713864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:35.708141    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" podUID="ab49f445-8557-460c-9080-8caa6e601004" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:01:35.799200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:35.799122    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" podStartSLOduration=2.799100324 podStartE2EDuration="2.799100324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:35.718342795 +0000 UTC m=+1708.297752187" watchObservedRunningTime="2026-11-02 18:01:35.799100324 +0000 UTC m=+1708.378509704"
Nov 02 18:01:36.502951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.502834    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" podUID="d9e28883-4069-480e-b9c4-58529708f5da" containerName="main" containerID="cri-o://c3a59077fdc42cf3fab05d77516f57e78c9a4a2c360298b2cd0e7e283d077192" gracePeriod=45
Nov 02 18:01:36.532602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.532551    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" podUID="d9e28883-4069-480e-b9c4-58529708f5da" containerName="main" probeResult="failure" output="Get \"https://10.132.0.206:8000/ready\": dial tcp 10.132.0.206:8000: connect: connection refused"
Nov 02 18:01:36.694863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.694307    5042 generic.go:356] "Generic (PLEG): container finished" podID="d9e28883-4069-480e-b9c4-58529708f5da" containerID="c3a59077fdc42cf3fab05d77516f57e78c9a4a2c360298b2cd0e7e283d077192" exitCode=0
Nov 02 18:01:36.699005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.695885    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" event={"ID":"d9e28883-4069-480e-b9c4-58529708f5da","Type":"ContainerDied","Data":"c3a59077fdc42cf3fab05d77516f57e78c9a4a2c360298b2cd0e7e283d077192"}
Nov 02 18:01:36.699005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.696731    5042 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tw9kl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:01:36.699005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.696789    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" podUID="ab49f445-8557-460c-9080-8caa6e601004" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:01:36.904634 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.904599    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 18:01:36.984556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.984519    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir\") pod \"d9e28883-4069-480e-b9c4-58529708f5da\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") "
Nov 02 18:01:36.984787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.984601    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv\" (UniqueName: \"kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv\") pod \"d9e28883-4069-480e-b9c4-58529708f5da\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") "
Nov 02 18:01:36.984787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.984678    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs\") pod \"d9e28883-4069-480e-b9c4-58529708f5da\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") "
Nov 02 18:01:36.984951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.984857    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home\") pod \"d9e28883-4069-480e-b9c4-58529708f5da\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") "
Nov 02 18:01:36.984951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.984827    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir" pod "d9e28883-4069-480e-b9c4-58529708f5da" (UID: "d9e28883-4069-480e-b9c4-58529708f5da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:36.985068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.984977    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm\") pod \"d9e28883-4069-480e-b9c4-58529708f5da\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") "
Nov 02 18:01:36.985128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.985059    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache\") pod \"d9e28883-4069-480e-b9c4-58529708f5da\" (UID: \"d9e28883-4069-480e-b9c4-58529708f5da\") "
Nov 02 18:01:36.985220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.985196    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home" pod "d9e28883-4069-480e-b9c4-58529708f5da" (UID: "d9e28883-4069-480e-b9c4-58529708f5da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:36.985432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.985399    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache" pod "d9e28883-4069-480e-b9c4-58529708f5da" (UID: "d9e28883-4069-480e-b9c4-58529708f5da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:36.985951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.985929    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:36.986148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.985954    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:36.986148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.985968    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:36.988142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.988116    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs" pod "d9e28883-4069-480e-b9c4-58529708f5da" (UID: "d9e28883-4069-480e-b9c4-58529708f5da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:36.988248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.988163    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv" pod "d9e28883-4069-480e-b9c4-58529708f5da" (UID: "d9e28883-4069-480e-b9c4-58529708f5da"). InnerVolumeSpecName "kube-api-access-bc2jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:36.990013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:36.989985    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm" pod "d9e28883-4069-480e-b9c4-58529708f5da" (UID: "d9e28883-4069-480e-b9c4-58529708f5da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:37.091990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.091946    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9e28883-4069-480e-b9c4-58529708f5da-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:37.096055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.096027    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc2jv\" (UniqueName: \"kubernetes.io/projected/d9e28883-4069-480e-b9c4-58529708f5da-kube-api-access-bc2jv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:37.096448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.096379    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9e28883-4069-480e-b9c4-58529708f5da-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:01:37.708073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.708036    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c" event={"ID":"d9e28883-4069-480e-b9c4-58529708f5da","Type":"ContainerDied","Data":"692d89444255941fe21aded4cbe20195c2960d18bab7a19853d11cd72a4232b9"}
Nov 02 18:01:37.708590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.708093    5042 scope.go:122] "RemoveContainer" containerID="c3a59077fdc42cf3fab05d77516f57e78c9a4a2c360298b2cd0e7e283d077192"
Nov 02 18:01:37.708755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.708730    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"
Nov 02 18:01:37.712201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.710449    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:01:37.734375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.734336    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"]
Nov 02 18:01:37.745993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:37.745957    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-b1a788/auth-test-model-b1a788-kserve-57d789d69-sfk5c"]
Nov 02 18:01:39.623602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:39.623563    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e28883-4069-480e-b9c4-58529708f5da" path="/var/lib/kubelet/pods/d9e28883-4069-480e-b9c4-58529708f5da/volumes"
Nov 02 18:01:41.564206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:41.564178    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:01:48.747478 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:01:48.747427    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:01:50.261228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:50.261168    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:50.315031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:50.314986    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:50.518671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:50.518574    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:50.527963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:50.519617    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 18:01:50.620520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:50.620477    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:51.094909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:51.094252    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:01:51.364965 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:51.362969    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:51.364965 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:51.363585    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:01:51.543995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:51.526999    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:51.543995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:51.527614    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:01:52.132908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.130697    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:01:52.156876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.136950    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:01:52.280317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.280271    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.282148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.282115    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.285476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.285433    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.369908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.369833    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.514058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.514011    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.589857 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.519040    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.590681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.546351    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:52.592299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.592267    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:01:52.593394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.592798    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:01:52.777277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:52.769913    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:01:53.138073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:53.137952    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:53.337877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:01:53.335947    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:02:02.861999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:02.861410    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:02:02.861999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:02.861491    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:04.029127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:04.029084    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"]
Nov 02 18:02:04.034393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:04.034358    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-936ac2e1-tcw9p"]
Nov 02 18:02:05.547502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:05.547467    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2047d9fb-6cc6-4e1e-885e-7d5732943848" path="/var/lib/kubelet/pods/2047d9fb-6cc6-4e1e-885e-7d5732943848/volumes"
Nov 02 18:02:07.818920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:07.818879    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"]
Nov 02 18:02:07.846039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:07.840748    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy" containerID="cri-o://849dd99cd9a9c862fcff142ce5109216f838cebf3860dc443f989a81579c6839" gracePeriod=30
Nov 02 18:02:09.095613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:09.095568    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"]
Nov 02 18:02:09.096135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:09.095919    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy" containerID="cri-o://9362b51fa34c8dbf29dfd73277f55119a3ca6b556d063e575a6b4ab4da1eac62" gracePeriod=30
Nov 02 18:02:09.626264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:09.626224    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"]
Nov 02 18:02:09.982766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:09.982037    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d9e28883-4069-480e-b9c4-58529708f5da" containerName="main"
Nov 02 18:02:09.982766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:09.982089    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e28883-4069-480e-b9c4-58529708f5da" containerName="main"
Nov 02 18:02:09.982766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:09.982511    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e28883-4069-480e-b9c4-58529708f5da" containerName="main"
Nov 02 18:02:10.095863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.091483    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"]
Nov 02 18:02:10.095863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.091648    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.138781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.138711    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5c3c1f-openshift-default-dockercfg-6bkwm\""
Nov 02 18:02:10.201683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.201448    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:02:10.201683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.201525    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:10.228056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.227766    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.227806    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228038    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228114    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228181    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228218    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228264    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvp4\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228293    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.228390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.228331    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.332734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.330880    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.332734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.330939    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337417    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337522    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337608    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337652    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337712    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvp4\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337750    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.337797    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.338657    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.338956    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.339425    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.339689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.339640    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.342894 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.341825    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.540828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.540794    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.541014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.540859    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.541014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.540943    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvp4\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.541124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.541106    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy\") pod \"e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") " pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:10.659944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:10.659868    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:12.052315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.052280    5042 scope.go:122] "RemoveContainer" containerID="87ef3ecbe2f482f9404dc05c2f0a74427ca846e6b5506ef43b9a40c04cdc6cc1"
Nov 02 18:02:12.644046 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.644011    5042 scope.go:122] "RemoveContainer" containerID="694c9254437c9f842255ea101b0b184b503e837a56accc14d4dff35d9063e91e"
Nov 02 18:02:12.687108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.687078    5042 scope.go:122] "RemoveContainer" containerID="7f9a5a451c8868fc8f9a41ecec252742f92b637c52160e11c65c71293ddd75d5"
Nov 02 18:02:12.734536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.734510    5042 scope.go:122] "RemoveContainer" containerID="f500c566e93b9ad02620af945ca5024495fd6e3dc4b1049a66a49631dab0b392"
Nov 02 18:02:12.794476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.794447    5042 scope.go:122] "RemoveContainer" containerID="b19cf4a6c3798edfc075a046cc51a45de674717119290db9f4bcb3ddb4dcb98f"
Nov 02 18:02:12.809194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.809171    5042 scope.go:122] "RemoveContainer" containerID="cc85c46d02598dc044b6b001ce182ef2acd860f5e2409de9c23ac6fff89d021b"
Nov 02 18:02:12.840129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:12.840100    5042 scope.go:122] "RemoveContainer" containerID="a41c1bcad832411c312e06da1fe5e05e9ee066a6016e9edc40fc5fe5bf7b4273"
Nov 02 18:02:13.239023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.238985    5042 generic.go:356] "Generic (PLEG): container finished" podID="68d62cc7-9fc5-455e-81a8-d71472720401" containerID="849dd99cd9a9c862fcff142ce5109216f838cebf3860dc443f989a81579c6839" exitCode=0
Nov 02 18:02:13.239523 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.239179    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" event={"ID":"68d62cc7-9fc5-455e-81a8-d71472720401","Type":"ContainerDied","Data":"849dd99cd9a9c862fcff142ce5109216f838cebf3860dc443f989a81579c6839"}
Nov 02 18:02:13.348307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.348276    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"]
Nov 02 18:02:13.485602 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:02:13.480384    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc8f6b1_4cf9_4715_8974_3cdf3a90cee6.slice/crio-673eed2ef67a8a7d125586af8b2fc0889ac708d988cd46bb343d52cc0443d715 WatchSource:0}: Error finding container 673eed2ef67a8a7d125586af8b2fc0889ac708d988cd46bb343d52cc0443d715: Status 404 returned error can't find the container with id 673eed2ef67a8a7d125586af8b2fc0889ac708d988cd46bb343d52cc0443d715
Nov 02 18:02:13.676091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.670528    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:13.676091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.670644    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:13.676091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.670690    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:13.855470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:13.855388    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.008954    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009048    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009103    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009190    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009283    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009342    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009522    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009587    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.011065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.009702    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy\") pod \"68d62cc7-9fc5-455e-81a8-d71472720401\" (UID: \"68d62cc7-9fc5-455e-81a8-d71472720401\") "
Nov 02 18:02:14.036297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.035809    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:14.036297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.035975    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:14.036565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.036338    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:14.036565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.036479    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:14.036796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.036692    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:14.039309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.039274    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:14.039309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.039691    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:02:14.040095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.039893    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:14.040592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.040567    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k" pod "68d62cc7-9fc5-455e-81a8-d71472720401" (UID: "68d62cc7-9fc5-455e-81a8-d71472720401"). InnerVolumeSpecName "kube-api-access-7446k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113673    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d62cc7-9fc5-455e-81a8-d71472720401-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113701    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d62cc7-9fc5-455e-81a8-d71472720401-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113713    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113723    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113737    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113747    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113756    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d62cc7-9fc5-455e-81a8-d71472720401-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113764    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.128416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.113772    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7446k\" (UniqueName: \"kubernetes.io/projected/68d62cc7-9fc5-455e-81a8-d71472720401-kube-api-access-7446k\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:14.259028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.258998    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"
Nov 02 18:02:14.259522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.259024    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc" event={"ID":"68d62cc7-9fc5-455e-81a8-d71472720401","Type":"ContainerDied","Data":"47cf1426476936e51b030469cabe426e1e65da6574a61f4edbc130bff4d3fc10"}
Nov 02 18:02:14.259522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.259103    5042 scope.go:122] "RemoveContainer" containerID="849dd99cd9a9c862fcff142ce5109216f838cebf3860dc443f989a81579c6839"
Nov 02 18:02:14.271214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.271118    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" event={"ID":"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6","Type":"ContainerStarted","Data":"673eed2ef67a8a7d125586af8b2fc0889ac708d988cd46bb343d52cc0443d715"}
Nov 02 18:02:14.310025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.309914    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"]
Nov 02 18:02:14.315812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:14.315784    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-x7pgc"]
Nov 02 18:02:15.557866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.531450    5042 generic.go:356] "Generic (PLEG): container finished" podID="d173258a-c30f-463b-8aa0-298885d64950" containerID="9362b51fa34c8dbf29dfd73277f55119a3ca6b556d063e575a6b4ab4da1eac62" exitCode=0
Nov 02 18:02:15.557866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.531605    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" event={"ID":"d173258a-c30f-463b-8aa0-298885d64950","Type":"ContainerDied","Data":"9362b51fa34c8dbf29dfd73277f55119a3ca6b556d063e575a6b4ab4da1eac62"}
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.559817    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570000    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570109    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570145    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570292    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570325    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570464    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570501    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570542    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.570589    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token\") pod \"d173258a-c30f-463b-8aa0-298885d64950\" (UID: \"d173258a-c30f-463b-8aa0-298885d64950\") "
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.572339    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.572560    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.572944    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.573939    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.574081    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:15.586912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.586383    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podStartSLOduration=6.586362939 podStartE2EDuration="6.586362939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:15.578996775 +0000 UTC m=+1748.158406488" watchObservedRunningTime="2026-11-02 18:02:15.586362939 +0000 UTC m=+1748.165772326"
Nov 02 18:02:15.587970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.587364    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:15.601938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.589963    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:02:15.601938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.592985    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:15.601938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.594167    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj" pod "d173258a-c30f-463b-8aa0-298885d64950" (UID: "d173258a-c30f-463b-8aa0-298885d64950"). InnerVolumeSpecName "kube-api-access-kvpzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:15.601938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.594307    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" path="/var/lib/kubelet/pods/68d62cc7-9fc5-455e-81a8-d71472720401/volumes"
Nov 02 18:02:15.601938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.596611    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" event={"ID":"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6","Type":"ContainerStarted","Data":"d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"}
Nov 02 18:02:15.662416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.661976    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:15.662416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.662102    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:02:15.662416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.662150    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674416    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674451    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvpzj\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-kube-api-access-kvpzj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674466    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674480    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674494    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674510    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d173258a-c30f-463b-8aa0-298885d64950-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674523    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d173258a-c30f-463b-8aa0-298885d64950-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674537    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d173258a-c30f-463b-8aa0-298885d64950-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:15.674758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:15.674551    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d173258a-c30f-463b-8aa0-298885d64950-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:02:16.562418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:16.558356    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff" event={"ID":"d173258a-c30f-463b-8aa0-298885d64950","Type":"ContainerDied","Data":"1424241ece05e806d34cb027e10bc4b1825226c85faf7d003e486e001047f539"}
Nov 02 18:02:16.562418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:16.558436    5042 scope.go:122] "RemoveContainer" containerID="9362b51fa34c8dbf29dfd73277f55119a3ca6b556d063e575a6b4ab4da1eac62"
Nov 02 18:02:16.562418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:16.558935    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"
Nov 02 18:02:16.594099 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:16.587413    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"]
Nov 02 18:02:16.596576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:16.596544    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-6jqff"]
Nov 02 18:02:16.665930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:16.665901    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:17.546932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:17.546901    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d173258a-c30f-463b-8aa0-298885d64950" path="/var/lib/kubelet/pods/d173258a-c30f-463b-8aa0-298885d64950/volumes"
Nov 02 18:02:17.569654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:17.569622    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:17.570727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:17.570705    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:02:17.856109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:17.856019    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:02:17.856280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:17.856129    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.461444    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"]
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.462610    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy"
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.462633    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy"
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.462660    5042 cpu_manager.go:414] "containerMap: removing container" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy"
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.462670    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy"
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.462894    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d173258a-c30f-463b-8aa0-298885d64950" containerName="istio-proxy"
Nov 02 18:02:19.470863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.462910    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d62cc7-9fc5-455e-81a8-d71472720401" containerName="istio-proxy"
Nov 02 18:02:19.559865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.554246    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"]
Nov 02 18:02:19.559865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.559032    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.575873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.571345    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5c3c1f-dockercfg-gshv2\""
Nov 02 18:02:19.575873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.571429    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5c3c1f\""
Nov 02 18:02:19.598870 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:02:19.596737    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:02:19.624275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.620976    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7c5g\" (UniqueName: \"kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g\") pod \"payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.624275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.621072    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.722953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.722868    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7c5g\" (UniqueName: \"kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g\") pod \"payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.723233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.723213    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.753212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.753182    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.762065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.762032    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"]
Nov 02 18:02:19.902068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.902011    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"]
Nov 02 18:02:19.902068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.902021    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:19.953573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.953542    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7c5g\" (UniqueName: \"kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g\") pod \"payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:19.953923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.953905    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:19.954126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:19.954108    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg867\" (UniqueName: \"kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867\") pod \"payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:20.056452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.056130    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:20.065863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.056924    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg867\" (UniqueName: \"kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867\") pod \"payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:20.065863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.057229    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:20.065863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.063359    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:20.080911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.080760    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg867\" (UniqueName: \"kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867\") pod \"payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:20.300356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.300133    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:20.513472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.507622    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"]
Nov 02 18:02:20.646593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.646558    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"]
Nov 02 18:02:20.675553 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:02:20.675520    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2638b1_ae0f_4bec_996b_1ccd5f40bb23.slice/crio-c11d859b6ce1d2a382e0f48e7b0be5d05c793bdbc6f64baf1d9a86d0afc245d6 WatchSource:0}: Error finding container c11d859b6ce1d2a382e0f48e7b0be5d05c793bdbc6f64baf1d9a86d0afc245d6: Status 404 returned error can't find the container with id c11d859b6ce1d2a382e0f48e7b0be5d05c793bdbc6f64baf1d9a86d0afc245d6
Nov 02 18:02:20.746638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.746603    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" event={"ID":"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23","Type":"ContainerStarted","Data":"c11d859b6ce1d2a382e0f48e7b0be5d05c793bdbc6f64baf1d9a86d0afc245d6"}
Nov 02 18:02:20.748547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.748515    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" event={"ID":"8154703c-cc6b-47c3-b46f-c38c3ef721cb","Type":"ContainerStarted","Data":"c9b7d40a5b7a21a0d880150abaf1d682e334d5889b3b66a69dcb08a613aa9aed"}
Nov 02 18:02:20.803107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.803077    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"]
Nov 02 18:02:20.933655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.933575    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"]
Nov 02 18:02:20.941271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.936327    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981109    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981155    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjlb\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981187    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981247    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981351    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981608    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981632    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981676    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:20.982294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:20.981769    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.083633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083598    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083653    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083702    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083751    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083880    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083911    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjlb\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.083942    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.084001    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.084053    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.084129    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.096925    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.097159    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.097406    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.097881    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.099994    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.103083    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.103402    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.103567    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjlb\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.425202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.321288    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:21.660286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.660253    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"]
Nov 02 18:02:21.661253 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:02:21.661223    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1ea13b_c308_408a_83fd_bc29c2240d18.slice/crio-1d6ee9e45e83e69dcc704c9ccff08cdb3dee2fceb1fde8dc6078988e11a9a9df WatchSource:0}: Error finding container 1d6ee9e45e83e69dcc704c9ccff08cdb3dee2fceb1fde8dc6078988e11a9a9df: Status 404 returned error can't find the container with id 1d6ee9e45e83e69dcc704c9ccff08cdb3dee2fceb1fde8dc6078988e11a9a9df
Nov 02 18:02:21.665428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.665368    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.665550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.665470    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.665550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.665504    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.759223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.759192    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" event={"ID":"8154703c-cc6b-47c3-b46f-c38c3ef721cb","Type":"ContainerStarted","Data":"f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085"}
Nov 02 18:02:21.759989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.759966    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:21.762718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.762680    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" event={"ID":"5f1ea13b-c308-408a-83fd-bc29c2240d18","Type":"ContainerStarted","Data":"1d6ee9e45e83e69dcc704c9ccff08cdb3dee2fceb1fde8dc6078988e11a9a9df"}
Nov 02 18:02:21.785864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.764893    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" event={"ID":"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23","Type":"ContainerStarted","Data":"d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8"}
Nov 02 18:02:21.785864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.765009    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:21.785864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.784668    5042 generic.go:356] "Generic (PLEG): container finished" podID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerID="d3193892c26145e1d67b700784c204cc8826561fa2977d79a46d78f815e12fad" exitCode=0
Nov 02 18:02:21.785864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.784873    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" event={"ID":"87c8e2ba-6618-44e0-80c0-7823fdfeba6a","Type":"ContainerDied","Data":"d3193892c26145e1d67b700784c204cc8826561fa2977d79a46d78f815e12fad"}
Nov 02 18:02:21.785864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.785578    5042 scope.go:122] "RemoveContainer" containerID="d3193892c26145e1d67b700784c204cc8826561fa2977d79a46d78f815e12fad"
Nov 02 18:02:21.790793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.790260    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.790793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.790423    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.790793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.790476    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.839219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.837288    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" podStartSLOduration=2.837264601 podStartE2EDuration="2.837264601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:21.786698968 +0000 UTC m=+1754.366108355" watchObservedRunningTime="2026-11-02 18:02:21.837264601 +0000 UTC m=+1754.416673987"
Nov 02 18:02:21.926060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:21.905795    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" podStartSLOduration=2.905781526 podStartE2EDuration="2.905781526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:21.86061417 +0000 UTC m=+1754.440023556" watchObservedRunningTime="2026-11-02 18:02:21.905781526 +0000 UTC m=+1754.485190911"
Nov 02 18:02:22.802862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:22.802137    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" event={"ID":"87c8e2ba-6618-44e0-80c0-7823fdfeba6a","Type":"ContainerStarted","Data":"1b5190212a0a4d5e9ee0472122e7528c6952bfb40078e8a4794aaf4cdccfb445"}
Nov 02 18:02:22.818415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:22.814796    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" event={"ID":"5f1ea13b-c308-408a-83fd-bc29c2240d18","Type":"ContainerStarted","Data":"d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688"}
Nov 02 18:02:22.851585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:22.851526    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podStartSLOduration=2.851511312 podStartE2EDuration="2.851511312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:22.850903807 +0000 UTC m=+1755.430313194" watchObservedRunningTime="2026-11-02 18:02:22.851511312 +0000 UTC m=+1755.430920698"
Nov 02 18:02:23.047508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:23.047468    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:02:23.047717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:23.047686    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:23.047794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:23.047746    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:23.324869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:23.323914    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:23.336867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:23.334393    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:23.336867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:23.334465    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:24.066252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:24.066213    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:24.067224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:24.067189    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:24.329918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:24.327352    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:24.329918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:24.327428    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:24.956614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:24.955239    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"]
Nov 02 18:02:25.377389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.229969    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:25.377389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.230043    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:25.377389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.305022    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:02:25.377389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.305093    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:25.377389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.322387    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:25.377389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.322439    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:25.658650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.655753    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.671484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.663988    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bee8f6-openshift-default-dockercfg-nxqnx\""
Nov 02 18:02:25.781868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.779005    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs554\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.786953    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787053    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787206    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787340    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787385    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787651    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787693    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.790884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.787730    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.910021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.909922    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.911755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.911725    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.927697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.927668    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.927917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.927901    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.928205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.928187    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs554\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.928349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.928336    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.928511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.928498    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.930318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.930297    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.930587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.930568    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.936339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.936316    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.961910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.947960    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.969311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.928968    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.969311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.910988    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.969558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.958884    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.969558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.964902    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.969558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.965953    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.970920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.970891    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.983868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.972070    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs554\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:25.983868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:25.974896    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"]
Nov 02 18:02:26.009474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.009191    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:26.095544 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.085684    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:26.095544 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.085746    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:26.330867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.326182    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:26.330867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.326258    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:26.847648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.847585    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"]
Nov 02 18:02:26.867712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.867670    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:26.867819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.867767    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:26.867819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:26.867810    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:27.052639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.052216    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:27.052639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.052279    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:27.341521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.340641    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:27.341521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.340702    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:27.930864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.929273    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" event={"ID":"8b63ef55-620f-40bc-acef-2939dbad9279","Type":"ContainerStarted","Data":"b21ac7e159f5a5e29d20dbf3cf7e6e30d5dab7fe31700009c79885b6d044984a"}
Nov 02 18:02:27.930864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.929341    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" event={"ID":"8b63ef55-620f-40bc-acef-2939dbad9279","Type":"ContainerStarted","Data":"af9119d57f519fa8c0ed2bf2da1e5e1d69c916ac70b673c7169598d730e0325f"}
Nov 02 18:02:27.935868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.933443    5042 generic.go:356] "Generic (PLEG): container finished" podID="c74beb22-3816-4661-aa11-2c034967e021" containerID="830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361" exitCode=0
Nov 02 18:02:27.935868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.933498    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" event={"ID":"c74beb22-3816-4661-aa11-2c034967e021","Type":"ContainerDied","Data":"830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"}
Nov 02 18:02:27.935868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.934108    5042 scope.go:122] "RemoveContainer" containerID="830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"
Nov 02 18:02:27.940259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.936908    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:27.940259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.937000    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:27.940259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:27.937050    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:28.048903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.019386    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podStartSLOduration=4.019364267 podStartE2EDuration="4.019364267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:27.967254625 +0000 UTC m=+1760.546664010" watchObservedRunningTime="2026-11-02 18:02:28.019364267 +0000 UTC m=+1760.598773654"
Nov 02 18:02:28.058243 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.038504    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:28.058243 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.038577    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 18:02:28.058243 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.058061    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 18:02:28.059490 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.058670    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:28.059751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.058717    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:28.367475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.365699    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:28.367475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:28.365760    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:29.136864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.125132    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:29.138338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.138285    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:29.222489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.220113    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" event={"ID":"c74beb22-3816-4661-aa11-2c034967e021","Type":"ContainerStarted","Data":"c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd"}
Nov 02 18:02:29.267007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.266963    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:29.290449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.274990    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:02:29.407655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.392105    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:29.407655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.392192    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:29.407655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.393465    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:29.407655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.393521    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:29.594209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.593944    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:02:29.596611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.594747    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:29.659164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.658688    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:29.874985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.867133    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Nov 02 18:02:29.919732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:29.892173    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:30.131263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.130194    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:30.287403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.285828    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:30.299986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.299950    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:30.300291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.300259    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:30.332865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.323186    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:30.332865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.323248    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:30.347051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.345060    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:30.347350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.347315    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:30.404365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.365309    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 18:02:30.615960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.612917    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:30.633874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.618248    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:02:30.752809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.752776    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:30.828639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.827203    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:02:30.908651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.908539    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:30.911136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.911098    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 18:02:30.911877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:30.911833    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:02:31.023964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.023920    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:31.024733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.024709    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:02:31.070868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.037945    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:31.070868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.040443    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 18:02:31.073751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.073710    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:31.074025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.073993    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:31.158923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.136122    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:31.160765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.160735    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:02:31.279444 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.279402    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:31.280170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.280134    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:31.307866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.307356    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:31.444648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.444619    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:02:31.465864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.458195    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:02:31.489961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.472190    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:31.489961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.472254    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:31.759930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:31.759555    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.027914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.024355    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.059866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.055976    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.095610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.095576    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:32.095973 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.095939    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:32.139981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.139932    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.231505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.219112    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:02:32.231505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.219208    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:02:32.264156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.264120    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:02:32.340408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.340303    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:02:32.378873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.345520    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded"
Nov 02 18:02:32.492868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.425725    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body=
Nov 02 18:02:32.492868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.425782    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused"
Nov 02 18:02:32.492868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.427355    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:32.492868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.427402    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:32.492868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.458160    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.492868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.490404    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.574866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.559630    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:32.891140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.891104    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:02:32.959868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:32.947931    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:02:33.066566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.066524    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:33.092869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.067954    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:33.154042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.153928    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:33.247866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.207978    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:02:33.317296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.317258    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:33.317598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.317550    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:33.358263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.358204    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:33.500876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.497822    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:02:33.500876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:33.498631    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:02:34.035951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.035893    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:34.052510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.052482    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:34.055596 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.055551    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:34.069473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.069439    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:02:34.069778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.069573    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body=
Nov 02 18:02:34.069778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.069628    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused"
Nov 02 18:02:34.070013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.069984    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 18:02:34.155346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.155157    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:02:34.155552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.155426    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 18:02:34.156610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.156576    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:02:34.262302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.262270    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:02:34.323198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.323100    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:34.323198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:34.323175    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:35.293865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:35.279226    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:35.293865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:35.279291    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:35.310905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:35.305172    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:02:35.315640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:35.315120    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:02:36.306861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:36.306420    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:36.306861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:36.306486    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:37.279866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:37.276244    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:37.279866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:37.276306    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:38.283398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:38.283350    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Nov 02 18:02:38.284039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:38.284004    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Nov 02 18:02:39.146858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:39.146807    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:02:39.281793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:39.281756    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:02:39.440670 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:39.440645    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:02:48.607256 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:02:48.607206    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:02:49.482318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.482275    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"]
Nov 02 18:02:49.493233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.493197    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.496157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.495463    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"]
Nov 02 18:02:49.498049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.498017    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"agc-cred-871c0b31c1f7\""
Nov 02 18:02:49.499323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.499302    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"routing-overlay\""
Nov 02 18:02:49.499458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.499369    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"payload-processing-external-model-e2e--72426c61-dockercfg-2zfsk\""
Nov 02 18:02:49.499534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.499470    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-1ef96f\""
Nov 02 18:02:49.499601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.499558    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"kube-root-ca.crt\""
Nov 02 18:02:49.499665 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.499652    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-1ef96f\"/\"openshift-service-ca.crt\""
Nov 02 18:02:49.648142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.648092    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.648674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.648348    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.648674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.648411    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfhn\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.648674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.648532    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.761921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.761218    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.761921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.761432    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.761921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.761498    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfhn\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.761921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.761573    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.770884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.770814    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.771742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.771715    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.821181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.821109    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:49.857866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:49.850637    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfhn\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn\") pod \"payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") " pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:50.219489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:50.218255    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:50.557857 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:02:50.557812    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb114a0ef_16a0_4431_8831_bc28d99a6ff0.slice/crio-d86b0561d26be3379bdfe1964f94b1ea48261be30cec0baf06ea0e250e283a7b WatchSource:0}: Error finding container d86b0561d26be3379bdfe1964f94b1ea48261be30cec0baf06ea0e250e283a7b: Status 404 returned error can't find the container with id d86b0561d26be3379bdfe1964f94b1ea48261be30cec0baf06ea0e250e283a7b
Nov 02 18:02:50.560038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:50.560013    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"]
Nov 02 18:02:50.695106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:50.694290    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd" event={"ID":"b114a0ef-16a0-4431-8831-bc28d99a6ff0","Type":"ContainerStarted","Data":"d86b0561d26be3379bdfe1964f94b1ea48261be30cec0baf06ea0e250e283a7b"}
Nov 02 18:02:51.706352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:51.706303    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd" event={"ID":"b114a0ef-16a0-4431-8831-bc28d99a6ff0","Type":"ContainerStarted","Data":"bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033"}
Nov 02 18:02:51.706795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:51.706387    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:02:51.729153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:51.729101    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd" podStartSLOduration=2.729086961 podStartE2EDuration="2.729086961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:51.724302668 +0000 UTC m=+1784.303712055" watchObservedRunningTime="2026-11-02 18:02:51.729086961 +0000 UTC m=+1784.308496346"
Nov 02 18:02:57.918880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.918825    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"]
Nov 02 18:02:57.924076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.924055    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.940463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.940433    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"]
Nov 02 18:02:57.983583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983549    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.983791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983666    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.983791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983703    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.983791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2dc\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.984039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983809    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.984039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983879    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.984039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983901    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.984039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983952    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:57.984039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:57.983975    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086333    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086447    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086475    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086570    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2dc\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086614    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086699    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086735    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.086804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086783    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.087004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086805    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.087004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.086886    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.087216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.087196    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.087504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.087473    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.087604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.087574    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.087818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.087799    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.090164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.090133    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.090420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.090398    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.095784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.095759    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.096309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.096288    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2dc\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc\") pod \"e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") " pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:58.252864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:58.252182    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:02:59.003155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.003124    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"]
Nov 02 18:02:59.003651 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:02:59.003623    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4357bcb6_34a0_4f39_955f_f179a2bbdfeb.slice/crio-0c6dcc7661da35493f1cb35f556dfd34a676512a0dcca8b40ca78ceb925438ea WatchSource:0}: Error finding container 0c6dcc7661da35493f1cb35f556dfd34a676512a0dcca8b40ca78ceb925438ea: Status 404 returned error can't find the container with id 0c6dcc7661da35493f1cb35f556dfd34a676512a0dcca8b40ca78ceb925438ea
Nov 02 18:02:59.007300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.007271    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:59.007422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.007331    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:59.007422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.007365    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:59.854348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.854307    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" event={"ID":"4357bcb6-34a0-4f39-955f-f179a2bbdfeb","Type":"ContainerStarted","Data":"9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46"}
Nov 02 18:02:59.854555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.854375    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" event={"ID":"4357bcb6-34a0-4f39-955f-f179a2bbdfeb","Type":"ContainerStarted","Data":"0c6dcc7661da35493f1cb35f556dfd34a676512a0dcca8b40ca78ceb925438ea"}
Nov 02 18:02:59.892660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:02:59.892597    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podStartSLOduration=2.892582782 podStartE2EDuration="2.892582782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:59.887197815 +0000 UTC m=+1792.466607201" watchObservedRunningTime="2026-11-02 18:02:59.892582782 +0000 UTC m=+1792.471992168"
Nov 02 18:03:00.257436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:00.257405    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:03:00.258607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:00.258516    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 18:03:00.258607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:00.258569    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 18:03:01.269868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:01.260334    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 18:03:01.269868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:01.260396    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 18:03:02.258900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:02.258861    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 18:03:02.259110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:02.258935    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 18:03:02.728166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:02.728138    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:03:03.257662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:03.257627    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 18:03:03.257921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:03.257696    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 18:03:04.257663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:04.257628    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 18:03:04.258136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:04.257692    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 18:03:05.257895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:05.257798    5042 patch_prober.go:29] interesting pod/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused" start-of-body=
Nov 02 18:03:05.258316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:05.257915    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.250:15021/healthz/ready\": dial tcp 10.132.0.250:15021: connect: connection refused"
Nov 02 18:03:05.957993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:05.957942    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"]
Nov 02 18:03:05.958458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:05.958428    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy" containerID="cri-o://1b5190212a0a4d5e9ee0472122e7528c6952bfb40078e8a4794aaf4cdccfb445" gracePeriod=30
Nov 02 18:03:05.960984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:05.960960    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"]
Nov 02 18:03:05.961300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:05.961253    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy" containerID="cri-o://9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46" gracePeriod=30
Nov 02 18:03:09.388573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:09.388536    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:03:09.392942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:09.392917    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:03:09.397886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:09.397831    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:03:09.401464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:09.401436    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:03:11.260091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.260005    5042 generic.go:356] "Generic (PLEG): container finished" podID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerID="1b5190212a0a4d5e9ee0472122e7528c6952bfb40078e8a4794aaf4cdccfb445" exitCode=0
Nov 02 18:03:11.260660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.260204    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" event={"ID":"87c8e2ba-6618-44e0-80c0-7823fdfeba6a","Type":"ContainerDied","Data":"1b5190212a0a4d5e9ee0472122e7528c6952bfb40078e8a4794aaf4cdccfb445"}
Nov 02 18:03:11.260753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.260714    5042 scope.go:122] "RemoveContainer" containerID="d3193892c26145e1d67b700784c204cc8826561fa2977d79a46d78f815e12fad"
Nov 02 18:03:11.527173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.527146    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731188    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731283    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731422    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731458    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731502    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.757867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731578    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.778110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.731655    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.778110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.758624    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket\") pod \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\" (UID: \"87c8e2ba-6618-44e0-80c0-7823fdfeba6a\") "
Nov 02 18:03:11.794568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.794118    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:11.795116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.795088    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:11.805135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.805088    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:11.814994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.814958    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:11.946410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:11.886262    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.067345    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.067384    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.067401    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.067416    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.067430    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.067481    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"]
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.068774    5042 cpu_manager.go:414] "containerMap: removing container" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy"
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.068798    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy"
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.068817    5042 cpu_manager.go:414] "containerMap: removing container" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy"
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.068825    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy"
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.069131    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy"
Nov 02 18:03:12.114623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.069150    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" containerName="istio-proxy"
Nov 02 18:03:12.120716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.120681    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "kube-api-access-kf6bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:12.201824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.169545    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6bg\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-kube-api-access-kf6bg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.201824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.172619    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.201824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.180812    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:12.201824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.201741    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"]
Nov 02 18:03:12.207483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.207418    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:12.217668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.217102    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo" pod "87c8e2ba-6618-44e0-80c0-7823fdfeba6a" (UID: "87c8e2ba-6618-44e0-80c0-7823fdfeba6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:12.217668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.217559    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-36a991-openshift-default-dockercfg-7xrzr\""
Nov 02 18:03:12.297082 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297035    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297105    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297143    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8mb\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297227    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297268    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297336    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297411    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.297610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297485    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.298074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.297666    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.298074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.298057    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.298195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.298077    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.298195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.298097    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87c8e2ba-6618-44e0-80c0-7823fdfeba6a-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:12.327137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.327090    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8" event={"ID":"87c8e2ba-6618-44e0-80c0-7823fdfeba6a","Type":"ContainerDied","Data":"3eac565d84e90db184bcb5a0480163a49a21e6477cd3b02da5ad1aada5326e95"}
Nov 02 18:03:12.327353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.327231    5042 scope.go:122] "RemoveContainer" containerID="1b5190212a0a4d5e9ee0472122e7528c6952bfb40078e8a4794aaf4cdccfb445"
Nov 02 18:03:12.329217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.329113    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"
Nov 02 18:03:12.400925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.400427    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.400925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.400662    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.400925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.400695    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.400925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.400725    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8mb\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.400925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.400821    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.412947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.410416    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.412947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.410574    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.412947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.410656    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.412947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.410728    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.412947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.410808    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.415640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.414290    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.415640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.414967    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.415640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.415426    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.416027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.415728    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"]
Nov 02 18:03:12.416027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.415965    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.416946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.416919    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.421783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.421567    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.423940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.423911    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-74sb8"]
Nov 02 18:03:12.426276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.426179    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.428762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.428730    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8mb\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb\") pod \"e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") " pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.558552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.558507    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:12.919789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.917991    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"]
Nov 02 18:03:12.919789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.918211    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:12.919789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.918296    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:12.919789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:12.918339    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:13.407903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:13.384765    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:03:13.528590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:13.528554    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" event={"ID":"aa2a4865-484f-4b9c-9b3a-9b340885ab7f","Type":"ContainerStarted","Data":"fd89b47f0fc086319224987559fe3abb65267b5bba3cacc0eb484e165862ea16"}
Nov 02 18:03:13.744633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:13.699244    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c8e2ba-6618-44e0-80c0-7823fdfeba6a" path="/var/lib/kubelet/pods/87c8e2ba-6618-44e0-80c0-7823fdfeba6a/volumes"
Nov 02 18:03:14.024499 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:03:14.024351    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4357bcb6_34a0_4f39_955f_f179a2bbdfeb.slice/crio-9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4357bcb6_34a0_4f39_955f_f179a2bbdfeb.slice/crio-conmon-9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:03:14.136173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.136098    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:03:14.246646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.246608    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.246823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.246670    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.246823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.246764    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.246967 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.246832    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.247035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247003    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.247102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247083    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.247255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247218    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.247375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247274    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247689    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert\") pod \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\" (UID: \"4357bcb6-34a0-4f39-955f-f179a2bbdfeb\") "
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247683    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247805    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247781    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.247989    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.248435    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.248709    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.248735    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.248753    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.248768    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.248818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.248782    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.252061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.252037    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:14.252384 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.252358    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "kube-api-access-nv2dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:14.260426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.260393    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:14.260589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.260573    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy" pod "4357bcb6-34a0-4f39-955f-f179a2bbdfeb" (UID: "4357bcb6-34a0-4f39-955f-f179a2bbdfeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:14.350489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.350452    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.350489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.350483    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.350489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.350494    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2dc\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-kube-api-access-nv2dc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.350730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.350505    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4357bcb6-34a0-4f39-955f-f179a2bbdfeb-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:14.516013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.515981    5042 scope.go:122] "RemoveContainer" containerID="bae7bdc271ec45700ee965b3a60594f9ecb9136ecb3ce7d02dd7cdfe85fff0a5"
Nov 02 18:03:14.539408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.539370    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" event={"ID":"aa2a4865-484f-4b9c-9b3a-9b340885ab7f","Type":"ContainerStarted","Data":"19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"}
Nov 02 18:03:14.541954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.541929    5042 generic.go:356] "Generic (PLEG): container finished" podID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerID="9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46" exitCode=0
Nov 02 18:03:14.542084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.541970    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"
Nov 02 18:03:14.542084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.541988    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" event={"ID":"4357bcb6-34a0-4f39-955f-f179a2bbdfeb","Type":"ContainerDied","Data":"9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46"}
Nov 02 18:03:14.542084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.542048    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z" event={"ID":"4357bcb6-34a0-4f39-955f-f179a2bbdfeb","Type":"ContainerDied","Data":"0c6dcc7661da35493f1cb35f556dfd34a676512a0dcca8b40ca78ceb925438ea"}
Nov 02 18:03:14.542084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.542076    5042 scope.go:122] "RemoveContainer" containerID="9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46"
Nov 02 18:03:14.559335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.559280    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:14.559827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.559774    5042 scope.go:122] "RemoveContainer" containerID="9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46"
Nov 02 18:03:14.560148 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:03:14.560119    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46\": container with ID starting with 9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46 not found: ID does not exist" containerID="9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46"
Nov 02 18:03:14.560272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.560153    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46"} err="failed to get container status \"9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46\": rpc error: code = NotFound desc = could not find container \"9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46\": container with ID starting with 9f5b46aa0343e8369685c68df41167a5400cff1211716b7ff7788885d906ec46 not found: ID does not exist"
Nov 02 18:03:14.566024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.565999    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:14.589876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.586158    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podStartSLOduration=6.58613858 podStartE2EDuration="6.58613858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:14.576310453 +0000 UTC m=+1807.155719840" watchObservedRunningTime="2026-11-02 18:03:14.58613858 +0000 UTC m=+1807.165547968"
Nov 02 18:03:14.636219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.636182    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"]
Nov 02 18:03:14.640922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:14.640891    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4ea582-openshift-default-9b47c7c49-dnt4z"]
Nov 02 18:03:15.548606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:15.548572    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" path="/var/lib/kubelet/pods/4357bcb6-34a0-4f39-955f-f179a2bbdfeb/volumes"
Nov 02 18:03:15.552687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:15.552659    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:15.553969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:15.553950    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:03:18.551655 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:03:18.551617    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:03:18.618863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:18.613056    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:03:18.618863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:18.613123    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:20.921307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.921268    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"]
Nov 02 18:03:20.923146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.923119    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy"
Nov 02 18:03:20.923322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.923308    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy"
Nov 02 18:03:20.923709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.923693    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4357bcb6-34a0-4f39-955f-f179a2bbdfeb" containerName="istio-proxy"
Nov 02 18:03:20.951674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.951533    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"]
Nov 02 18:03:20.951865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.951687    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.995525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.995491    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.995885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.995858    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.996095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.996077    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.996281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.996262    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.996593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.996574    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.996688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.996608    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.996757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.996744    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq7kc\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.996951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.996913    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:20.997087 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:20.997002    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.098736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.098695    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.098811    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.098833    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.098898    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq7kc\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.098985    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099022    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099083    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099130    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099179    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099272    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099295    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099852    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.099997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.099976    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.100187 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.100069    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.101878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.101854    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.102887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.102813    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.109612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.109591    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq7kc\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.109745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.109616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token\") pod \"e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") " pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:21.299152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:21.298471    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:22.094553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.094519    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"]
Nov 02 18:03:22.101801 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:03:22.101754    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ba18d0_f9b6_4c59_823c_3cda2b2dd7a8.slice/crio-d0e782e46aec15fdfc314a8bb973aa73d24fa54a06171c983f713b487dff343c WatchSource:0}: Error finding container d0e782e46aec15fdfc314a8bb973aa73d24fa54a06171c983f713b487dff343c: Status 404 returned error can't find the container with id d0e782e46aec15fdfc314a8bb973aa73d24fa54a06171c983f713b487dff343c
Nov 02 18:03:22.106044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.106002    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:22.106186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.106099    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:22.106186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.106142    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:22.863225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.863125    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" event={"ID":"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8","Type":"ContainerStarted","Data":"2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c"}
Nov 02 18:03:22.865096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.863225    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" event={"ID":"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8","Type":"ContainerStarted","Data":"d0e782e46aec15fdfc314a8bb973aa73d24fa54a06171c983f713b487dff343c"}
Nov 02 18:03:22.894330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:22.894267    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podStartSLOduration=2.894248685 podStartE2EDuration="2.894248685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:22.886274241 +0000 UTC m=+1815.465683628" watchObservedRunningTime="2026-11-02 18:03:22.894248685 +0000 UTC m=+1815.473658073"
Nov 02 18:03:23.303178 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.303134    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:23.303764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.303342    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:23.303764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.303388    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:23.928872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.928461    5042 generic.go:356] "Generic (PLEG): container finished" podID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerID="d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234" exitCode=0
Nov 02 18:03:23.929476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.929454    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" event={"ID":"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6","Type":"ContainerDied","Data":"d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"}
Nov 02 18:03:23.929932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.929914    5042 scope.go:122] "RemoveContainer" containerID="d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"
Nov 02 18:03:23.931828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.931794    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:23.931949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.931908    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:23.932019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:23.931964    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:24.309863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:24.305357    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:24.309863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:24.305484    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:24.945767 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:24.945721    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" event={"ID":"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6","Type":"ContainerStarted","Data":"6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124"}
Nov 02 18:03:25.314923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:25.303294    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:25.314923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:25.303357    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:25.664780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:25.661875    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:03:25.664780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:25.662034    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:25.664780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:25.662077    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:26.304866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:26.304692    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:26.304866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:26.304763    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:26.661442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:26.661292    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:26.662213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:26.662061    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:27.309864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:27.301218    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:27.309864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:27.301281    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:27.660961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:27.660354    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:27.660961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:27.660402    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:28.357866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:28.352178    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:28.357866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:28.352270    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:28.668933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:28.666896    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:28.668933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:28.667005    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:29.328255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:29.325967    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Nov 02 18:03:29.328255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:29.326047    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Nov 02 18:03:29.664936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:29.664831    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:29.665426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:29.664934    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:30.490865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:30.490186    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:30.491864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:30.491820    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:30.498033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:30.497409    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:03:30.660626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:30.660588    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:30.660975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:30.660942    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:31.660583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:31.660542    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body=
Nov 02 18:03:31.661055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:31.660603    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused"
Nov 02 18:03:32.570231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:32.570184    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:03:32.666236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:32.666198    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:03:33.483992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:33.483965    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:03:38.889926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:38.889886    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"]
Nov 02 18:03:38.890415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:38.890168    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy" containerID="cri-o://bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4" gracePeriod=30
Nov 02 18:03:38.898114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:38.898082    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"]
Nov 02 18:03:38.898365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:38.898319    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy" containerID="cri-o://f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e" gracePeriod=30
Nov 02 18:03:44.262366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.262331    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 18:03:44.301373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.300745    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.301373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.300971    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.301373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.301105    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.301373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.301153    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.302299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.302272    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.302416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.302292    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.302517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.302433    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.302898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.302818    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.303267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.303158    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:44.303356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.303302    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.303600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.303577    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.303674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.303647    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.305475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.305449    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket\") pod \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\" (UID: \"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7\") "
Nov 02 18:03:44.305710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.305689    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.306549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.306525    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.306549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.306547    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.306697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.306559    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.306697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.306569    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.306697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.306580    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.307152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.307124    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "kube-api-access-zwmqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:44.315451 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.315416    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.315625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.315605    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:44.316395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.316357    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"]
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.317672    5042 cpu_manager.go:414] "containerMap: removing container" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy"
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.317695    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy"
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.317986    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerName="istio-proxy"
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.322523    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo" pod "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" (UID: "20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.323923    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.324136    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"]
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.328229    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.328371    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.328523    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.328620    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-36a991-serving-cert\""
Nov 02 18:03:44.330491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.328677    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tqgzn\""
Nov 02 18:03:44.334016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.333989    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 18:03:44.407693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.407648    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.407897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.407734    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.407897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.407779    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.407947    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408082    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408154    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408231    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408314    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408462 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408432    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.408545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408448    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo\") pod \"048057a0-2665-4987-91ce-c40f6a5a1f24\" (UID: \"048057a0-2665-4987-91ce-c40f6a5a1f24\") "
Nov 02 18:03:44.408600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408567    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.408900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.408869    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88sc\" (UniqueName: \"kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.409077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409023    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.409222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409203    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.409421 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409401    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.409716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409696    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.409716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409703    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.409913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409722    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.409913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409737    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.409913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409752    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmqf\" (UniqueName: \"kubernetes.io/projected/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-kube-api-access-zwmqf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.409913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409766    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.409913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.409779    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.410207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.410084    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.410574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.410552    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:03:44.413514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.413483    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:44.413772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.413750    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:03:44.416037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.415832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:03:44.416157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.416113    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6" pod "048057a0-2665-4987-91ce-c40f6a5a1f24" (UID: "048057a0-2665-4987-91ce-c40f6a5a1f24"). InnerVolumeSpecName "kube-api-access-ndwd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:03:44.511714 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.511674    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.511926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.511860    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.512023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512006    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.512179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512163    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88sc\" (UniqueName: \"kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.512273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512254    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512270    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512284    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048057a0-2665-4987-91ce-c40f6a5a1f24-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512302    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndwd6\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-kube-api-access-ndwd6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512316    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512330    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048057a0-2665-4987-91ce-c40f6a5a1f24-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512346    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048057a0-2665-4987-91ce-c40f6a5a1f24-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:03:44.512658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.512616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.514723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.514700    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.514907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.514884    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.521241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.521213    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88sc\" (UniqueName: \"kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc\") pod \"maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:44.647276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:44.647238    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:45.169921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.169787    5042 generic.go:356] "Generic (PLEG): container finished" podID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" containerID="bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4" exitCode=0
Nov 02 18:03:45.177004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.170301    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" event={"ID":"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7","Type":"ContainerDied","Data":"bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4"}
Nov 02 18:03:45.177004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.170390    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg" event={"ID":"20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7","Type":"ContainerDied","Data":"4e25d8901c40386efd2d917d50ad61185145e42302f8faf038a15d8112546197"}
Nov 02 18:03:45.177004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.170451    5042 scope.go:122] "RemoveContainer" containerID="bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4"
Nov 02 18:03:45.177004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.170718    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"
Nov 02 18:03:45.198474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.197121    5042 generic.go:356] "Generic (PLEG): container finished" podID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerID="f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e" exitCode=0
Nov 02 18:03:45.198474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.197181    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" event={"ID":"048057a0-2665-4987-91ce-c40f6a5a1f24","Type":"ContainerDied","Data":"f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e"}
Nov 02 18:03:45.198474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.197227    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff" event={"ID":"048057a0-2665-4987-91ce-c40f6a5a1f24","Type":"ContainerDied","Data":"53e72060009589a40acbc910c7182f797c8bb92c46f13cc5b1e82351fdc8d56c"}
Nov 02 18:03:45.198474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.197302    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"
Nov 02 18:03:45.231559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.231531    5042 scope.go:122] "RemoveContainer" containerID="bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4"
Nov 02 18:03:45.233179 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:03:45.233143    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4\": container with ID starting with bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4 not found: ID does not exist" containerID="bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4"
Nov 02 18:03:45.233491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.233192    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4"} err="failed to get container status \"bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4\": rpc error: code = NotFound desc = could not find container \"bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4\": container with ID starting with bbabbd8b6be23eb7dc60a1000ede87d0b9a2a029c43dc7663b375282e8974ed4 not found: ID does not exist"
Nov 02 18:03:45.233491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.233218    5042 scope.go:122] "RemoveContainer" containerID="f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e"
Nov 02 18:03:45.236008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.235982    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"]
Nov 02 18:03:45.242710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.242669    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-nrvdg"]
Nov 02 18:03:45.254911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.253904    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"]
Nov 02 18:03:45.254911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.254297    5042 scope.go:122] "RemoveContainer" containerID="f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e"
Nov 02 18:03:45.254911 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:03:45.254721    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e\": container with ID starting with f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e not found: ID does not exist" containerID="f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e"
Nov 02 18:03:45.254911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.254756    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e"} err="failed to get container status \"f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e\": rpc error: code = NotFound desc = could not find container \"f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e\": container with ID starting with f6d72ebbcd7994c3083ab862ba6fb43c5ccb7f199cefaca84ab5030369ecfb0e not found: ID does not exist"
Nov 02 18:03:45.260249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.260225    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-79sff"]
Nov 02 18:03:45.370102 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:03:45.370065    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df0e107_1ccd_4e19_b8b0_cdbf95ef9d13.slice/crio-61d7fbd61a7d8a5a30c4991805ee38f3c71c49f5c3b9fe8df3c9a23f94b82811 WatchSource:0}: Error finding container 61d7fbd61a7d8a5a30c4991805ee38f3c71c49f5c3b9fe8df3c9a23f94b82811: Status 404 returned error can't find the container with id 61d7fbd61a7d8a5a30c4991805ee38f3c71c49f5c3b9fe8df3c9a23f94b82811
Nov 02 18:03:45.373893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.372099    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:03:45.374893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.374861    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"]
Nov 02 18:03:45.550491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.550452    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" path="/var/lib/kubelet/pods/048057a0-2665-4987-91ce-c40f6a5a1f24/volumes"
Nov 02 18:03:45.551361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:45.551335    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7" path="/var/lib/kubelet/pods/20b66dbd-84d1-470b-b0f8-2e51fd7a2ca7/volumes"
Nov 02 18:03:46.540864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:46.529315    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv" event={"ID":"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13","Type":"ContainerStarted","Data":"61d7fbd61a7d8a5a30c4991805ee38f3c71c49f5c3b9fe8df3c9a23f94b82811"}
Nov 02 18:03:48.669943 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:03:48.666221    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:03:52.931445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:52.931399    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv" event={"ID":"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13","Type":"ContainerStarted","Data":"758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167"}
Nov 02 18:03:52.931983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:52.931497    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:52.951936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:52.951723    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv" podStartSLOduration=9.887855199 podStartE2EDuration="16.951685176s" totalImagesPullingTime="7.063829977s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:52.951321684 +0000 UTC m=+1845.530731071" watchObservedRunningTime="2026-11-02 18:03:52.951685176 +0000 UTC m=+1845.531094563"
Nov 02 18:03:55.177028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.176988    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"]
Nov 02 18:03:55.178458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.178429    5042 cpu_manager.go:414] "containerMap: removing container" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy"
Nov 02 18:03:55.178458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.178455    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy"
Nov 02 18:03:55.178688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.178617    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="048057a0-2665-4987-91ce-c40f6a5a1f24" containerName="istio-proxy"
Nov 02 18:03:55.205337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.204291    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"]
Nov 02 18:03:55.205337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.204427    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.234637    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.236398    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.236580    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.236684    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.236778    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2fc\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.236815    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.236918    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.237006    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.241903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.237111    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.338799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.338756    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.338799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.338804    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.338888    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.338933    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.338998    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2fc\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339030    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339126    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339252    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339369    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339696    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339697    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339755    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.339993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.339872    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.341539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.341516    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.342004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.341984    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.348347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.348327    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.348684 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.348664    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2fc\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.543759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.543708    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:55.901525 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:03:55.901490    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa199220_32ef_4eea_a688_05f166f14d39.slice/crio-182d1b29a1cc9e33be29cb9354099b8f4d5c945cdc53107c2e361a78a5ec38da WatchSource:0}: Error finding container 182d1b29a1cc9e33be29cb9354099b8f4d5c945cdc53107c2e361a78a5ec38da: Status 404 returned error can't find the container with id 182d1b29a1cc9e33be29cb9354099b8f4d5c945cdc53107c2e361a78a5ec38da
Nov 02 18:03:55.904752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.904717    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"]
Nov 02 18:03:55.905109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.905076    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:55.905212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.905159    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:55.905212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.905206    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:03:55.963705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:55.963672    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" event={"ID":"aa199220-32ef-4eea-a688-05f166f14d39","Type":"ContainerStarted","Data":"182d1b29a1cc9e33be29cb9354099b8f4d5c945cdc53107c2e361a78a5ec38da"}
Nov 02 18:03:57.014865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:57.014357    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" event={"ID":"aa199220-32ef-4eea-a688-05f166f14d39","Type":"ContainerStarted","Data":"6873572cd2f7d3b465cc055bc615a163e703bad331990bfd253a0cf43e5a0f9b"}
Nov 02 18:03:57.077865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:57.072261    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podStartSLOduration=2.072241507 podStartE2EDuration="2.072241507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:57.066187079 +0000 UTC m=+1849.645596464" watchObservedRunningTime="2026-11-02 18:03:57.072241507 +0000 UTC m=+1849.651650897"
Nov 02 18:03:57.545733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:57.545694    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 18:03:57.545969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:57.545755    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 18:03:57.546483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:57.546463    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:03:58.544182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:58.544147    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 18:03:58.544604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:58.544203    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 18:03:58.947163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:58.947133    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:03:59.544597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:59.544558    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 18:03:59.545890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:03:59.544647    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 18:04:00.049735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.049704    5042 generic.go:356] "Generic (PLEG): container finished" podID="8b63ef55-620f-40bc-acef-2939dbad9279" containerID="b21ac7e159f5a5e29d20dbf3cf7e6e30d5dab7fe31700009c79885b6d044984a" exitCode=0
Nov 02 18:04:00.049982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.049772    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" event={"ID":"8b63ef55-620f-40bc-acef-2939dbad9279","Type":"ContainerDied","Data":"b21ac7e159f5a5e29d20dbf3cf7e6e30d5dab7fe31700009c79885b6d044984a"}
Nov 02 18:04:00.050320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.050305    5042 scope.go:122] "RemoveContainer" containerID="b21ac7e159f5a5e29d20dbf3cf7e6e30d5dab7fe31700009c79885b6d044984a"
Nov 02 18:04:00.052337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.052311    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:04:00.052439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.052383    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:04:00.052439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.052415    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:04:00.545866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.545593    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 18:04:00.545866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.545655    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 18:04:00.751166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:00.751130    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:04:01.060633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:01.060594    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" event={"ID":"8b63ef55-620f-40bc-acef-2939dbad9279","Type":"ContainerStarted","Data":"a9d28fa02ee73fbae4993a6b58af4f3851ad467579cc8f0d1dbe422c4eb987d3"}
Nov 02 18:04:01.544171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:01.544134    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 18:04:01.544360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:01.544212    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 18:04:02.036897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:02.023265    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:04:02.036897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:02.023394    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 18:04:02.036897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:02.023439    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 18:04:02.544478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:02.544440    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Nov 02 18:04:02.544675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:02.544495    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Nov 02 18:04:03.014587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:03.014544    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 18:04:03.014794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:03.014613    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 18:04:03.566268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:03.566233    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:04:03.566948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:03.566926    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:04:03.567730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:03.567706    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:04:04.015056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:04.015013    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 18:04:04.015280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:04.015068    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 18:04:05.015032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:05.015000    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 18:04:05.015500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:05.015056    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 18:04:06.014304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:06.014266    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 18:04:06.014511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:06.014336    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 18:04:07.019615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:07.019574    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:04:07.020175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:07.019989    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:04:07.020693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:07.020669    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:04:18.541591 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:04:18.541551    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:04:31.594022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:31.593977    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:31.594529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:31.594053    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:34.100392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:34.094569    5042 generic.go:356] "Generic (PLEG): container finished" podID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerID="19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0" exitCode=0
Nov 02 18:04:34.100392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:34.094949    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" event={"ID":"aa2a4865-484f-4b9c-9b3a-9b340885ab7f","Type":"ContainerDied","Data":"19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"}
Nov 02 18:04:34.100392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:34.095540    5042 scope.go:122] "RemoveContainer" containerID="19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"
Nov 02 18:04:34.107189 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:34.107142    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:04:34.107377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:34.107351    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:04:34.107473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:34.107456    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:04:35.117733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:35.117687    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" event={"ID":"aa2a4865-484f-4b9c-9b3a-9b340885ab7f","Type":"ContainerStarted","Data":"20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e"}
Nov 02 18:04:35.558944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:35.558918    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:04:35.559149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:35.559093    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 02 18:04:35.559227 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:35.559160    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 02 18:04:36.559925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:36.559884    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 02 18:04:36.560357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:36.559963    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 02 18:04:37.559470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:37.559430    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 02 18:04:37.559688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:37.559489    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 02 18:04:38.559235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:38.559199    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 02 18:04:38.559637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:38.559256    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 02 18:04:39.559115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:39.559079    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 02 18:04:39.559305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:39.559138    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 02 18:04:40.559757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:40.559723    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body=
Nov 02 18:04:40.560270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:40.559788    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused"
Nov 02 18:04:41.564120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:41.564087    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:04:41.564783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:41.564670    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:04:41.566150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:41.566119    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:04:48.543982 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:04:48.543942    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:04:51.610899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.610862    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"]
Nov 02 18:04:51.617549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.617519    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.627780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.622940    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"]
Nov 02 18:04:51.627780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.623688    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:04:51.627780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.624620    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:04:51.697414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.697367    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2g69\" (UniqueName: \"kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.697821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.697797    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.698109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.698090    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.845741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.827757    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2g69\" (UniqueName: \"kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.845741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.827942    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.845741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.828035    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.845741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.843058    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.845741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.843463    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.900958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.893919    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2g69\" (UniqueName: \"kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69\") pod \"maas-api-revoke-sub-82b13d41-5cqxc\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:51.949922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:51.949806    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:52.259307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:52.259263    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"]
Nov 02 18:04:52.264557 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:04:52.264526    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83aa7f6_4079_4822_b6b3_bd1b3ea69d09.slice/crio-a2337b8adf820ffb4d9b840454941bdc54ef1bbf36df67d4fee2ee77f179cc71 WatchSource:0}: Error finding container a2337b8adf820ffb4d9b840454941bdc54ef1bbf36df67d4fee2ee77f179cc71: Status 404 returned error can't find the container with id a2337b8adf820ffb4d9b840454941bdc54ef1bbf36df67d4fee2ee77f179cc71
Nov 02 18:04:52.942571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:52.942538    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc" event={"ID":"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09","Type":"ContainerStarted","Data":"1458d65198b98e0e1f9faefc64cf49d205a5f0df71a4425ac5689ab8df8eebf6"}
Nov 02 18:04:52.943002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:52.942591    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc" event={"ID":"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09","Type":"ContainerStarted","Data":"a2337b8adf820ffb4d9b840454941bdc54ef1bbf36df67d4fee2ee77f179cc71"}
Nov 02 18:04:52.966653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:52.966605    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc" podStartSLOduration=1.966589578 podStartE2EDuration="1.966589578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:52.964509674 +0000 UTC m=+1905.543919061" watchObservedRunningTime="2026-11-02 18:04:52.966589578 +0000 UTC m=+1905.545998958"
Nov 02 18:04:53.953092 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:53.953063    5042 generic.go:356] "Generic (PLEG): container finished" podID="f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" containerID="1458d65198b98e0e1f9faefc64cf49d205a5f0df71a4425ac5689ab8df8eebf6" exitCode=0
Nov 02 18:04:53.953521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:53.953152    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc" event={"ID":"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09","Type":"ContainerDied","Data":"1458d65198b98e0e1f9faefc64cf49d205a5f0df71a4425ac5689ab8df8eebf6"}
Nov 02 18:04:54.487763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:54.487717    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"]
Nov 02 18:04:54.488366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:54.488331    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" containerID="cri-o://70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c" gracePeriod=30
Nov 02 18:04:54.709120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:54.709081    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:04:54.709325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:54.709165    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:04:55.164517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.164484    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:55.331208 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.331119    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69\" (UniqueName: \"kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69\") pod \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") "
Nov 02 18:04:55.331403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.331236    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca\") pod \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") "
Nov 02 18:04:55.331478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.331460    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca\") pod \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\" (UID: \"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09\") "
Nov 02 18:04:55.331584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.331552    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca" pod "f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" (UID: "f83aa7f6-4079-4822-b6b3-bd1b3ea69d09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:04:55.332279 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.332232    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:04:55.333541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.333520    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69" pod "f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" (UID: "f83aa7f6-4079-4822-b6b3-bd1b3ea69d09"). InnerVolumeSpecName "kube-api-access-j2g69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:04:55.333808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.333787    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca" pod "f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" (UID: "f83aa7f6-4079-4822-b6b3-bd1b3ea69d09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:04:55.433871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.433819    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:04:55.433871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.433871    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2g69\" (UniqueName: \"kubernetes.io/projected/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09-kube-api-access-j2g69\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:04:55.971676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.971644    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"
Nov 02 18:04:55.971911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.971682    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc" event={"ID":"f83aa7f6-4079-4822-b6b3-bd1b3ea69d09","Type":"ContainerDied","Data":"a2337b8adf820ffb4d9b840454941bdc54ef1bbf36df67d4fee2ee77f179cc71"}
Nov 02 18:04:55.971911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:55.971729    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2337b8adf820ffb4d9b840454941bdc54ef1bbf36df67d4fee2ee77f179cc71"
Nov 02 18:04:57.323877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.322965    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"]
Nov 02 18:04:57.325425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.324399    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" containerName="revoke-keys"
Nov 02 18:04:57.325425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.324429    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" containerName="revoke-keys"
Nov 02 18:04:57.325425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.324652    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" containerName="revoke-keys"
Nov 02 18:04:57.350870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.348917    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.350870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.350582    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"]
Nov 02 18:04:57.355110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.355001    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:04:57.355303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.355170    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:04:57.381868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.379147    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.381868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.379277    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.381868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.379405    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn67r\" (UniqueName: \"kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.481272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.481233    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.481481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.481420    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn67r\" (UniqueName: \"kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.481656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.481637    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.482150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.482117    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.484291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.484266    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.489980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.489956    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn67r\" (UniqueName: \"kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r\") pod \"maas-api-revoke-sub-058b8c50-8kjgk\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:57.703480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:57.703446    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:04:58.025124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:58.025083    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"]
Nov 02 18:04:59.002374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.002344    5042 generic.go:356] "Generic (PLEG): container finished" podID="3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" containerID="201fcc84ead064dda8ad18d7aeb39de538e706975b5b06782b0574d3185d3ef0" exitCode=0
Nov 02 18:04:59.002708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.002431    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk" event={"ID":"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef","Type":"ContainerDied","Data":"201fcc84ead064dda8ad18d7aeb39de538e706975b5b06782b0574d3185d3ef0"}
Nov 02 18:04:59.002708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.002505    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk" event={"ID":"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef","Type":"ContainerStarted","Data":"a40f7139a2bc14e0744a32cfb20426b5e18314f20863b736752d29a71c2d174b"}
Nov 02 18:04:59.906122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.906090    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 18:04:59.950666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.950631    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"]
Nov 02 18:04:59.953024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.951933    5042 cpu_manager.go:414] "containerMap: removing container" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy"
Nov 02 18:04:59.953024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.951958    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy"
Nov 02 18:04:59.953024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.952241    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerName="istio-proxy"
Nov 02 18:04:59.965017 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.964987    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:04:59.966771 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.966742    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"]
Nov 02 18:04:59.967197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967031    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967107    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967176    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967293    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967349    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967392    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967636    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967703    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.967960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.967937    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy\") pod \"038c55b6-2933-42e9-a931-3e9c9e6210c0\" (UID: \"038c55b6-2933-42e9-a931-3e9c9e6210c0\") "
Nov 02 18:04:59.969362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.968450    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:04:59.969362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.969036    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:04:59.969362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.969165    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:04:59.969362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.969284    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:04:59.969651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.969601    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:04:59.981159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.970260    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:04:59.981159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.979263    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:04:59.981159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.979278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:04:59.981159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.979343    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:04:59.981159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:04:59.981003    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd" pod "038c55b6-2933-42e9-a931-3e9c9e6210c0" (UID: "038c55b6-2933-42e9-a931-3e9c9e6210c0"). InnerVolumeSpecName "kube-api-access-kwtsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:00.022555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.022509    5042 generic.go:356] "Generic (PLEG): container finished" podID="038c55b6-2933-42e9-a931-3e9c9e6210c0" containerID="70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c" exitCode=0
Nov 02 18:05:00.023009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.022569    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"
Nov 02 18:05:00.023009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.022854    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" event={"ID":"038c55b6-2933-42e9-a931-3e9c9e6210c0","Type":"ContainerDied","Data":"70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c"}
Nov 02 18:05:00.023009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.022914    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn" event={"ID":"038c55b6-2933-42e9-a931-3e9c9e6210c0","Type":"ContainerDied","Data":"998e7bf78aeac41128a46d96120af2bc30bbfc4fa3384756814b0b914717cab4"}
Nov 02 18:05:00.023009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.022951    5042 scope.go:122] "RemoveContainer" containerID="70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.070478    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfdfl\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.070619    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.070922    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.070967    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.070992    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071067    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071133    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071196    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071221    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071342    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071359    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071375    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071392    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071409    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtsd\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-kube-api-access-kwtsd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071423    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071438    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038c55b6-2933-42e9-a931-3e9c9e6210c0-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.073918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.071454    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038c55b6-2933-42e9-a931-3e9c9e6210c0-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.075902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.075878    5042 scope.go:122] "RemoveContainer" containerID="70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c"
Nov 02 18:05:00.076476 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:05:00.076405    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c\": container with ID starting with 70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c not found: ID does not exist" containerID="70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c"
Nov 02 18:05:00.076548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.076489    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c"} err="failed to get container status \"70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c\": rpc error: code = NotFound desc = could not find container \"70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c\": container with ID starting with 70a53ec50c977295a2f74051dfa24eeb76165c6aba07775a06f94f4d5aa1c44c not found: ID does not exist"
Nov 02 18:05:00.077107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.077079    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"]
Nov 02 18:05:00.083751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.083722    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-d95tn"]
Nov 02 18:05:00.172826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.172796    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173065    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173115    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173142    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173210    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173274    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173331    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173358    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173428    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfdfl\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.173673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173621    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.174241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.173969    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.174988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.174961    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.175129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.175028    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.175293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.175272    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.176884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.176859    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.177979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.177958    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.186458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.186440    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.192001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.191974    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfdfl\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl\") pod \"e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") " pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.235045 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.235020    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:05:00.323752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.323709    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:00.433911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.429992    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca\") pod \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") "
Nov 02 18:05:00.433911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.430066    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r\" (UniqueName: \"kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r\") pod \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") "
Nov 02 18:05:00.433911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.430137    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca\") pod \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\" (UID: \"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef\") "
Nov 02 18:05:00.433911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.431629    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca" pod "3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" (UID: "3977ce17-d1a7-4a34-a3a0-ddfb71a104ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:00.446757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.446711    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r" pod "3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" (UID: "3977ce17-d1a7-4a34-a3a0-ddfb71a104ef"). InnerVolumeSpecName "kube-api-access-xn67r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:00.448709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.448661    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca" pod "3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" (UID: "3977ce17-d1a7-4a34-a3a0-ddfb71a104ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:05:00.532735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.532698    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn67r\" (UniqueName: \"kubernetes.io/projected/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-kube-api-access-xn67r\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.533004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.532988    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.533106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.533095    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:00.726686 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:05:00.726652    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1159bd_e7f6_4ba4_9c5e_b1e66e174f78.slice/crio-8d66b2ae19d85dcbf4a0dc4901792053a3daf76968354dcb13bebd6325751b7e WatchSource:0}: Error finding container 8d66b2ae19d85dcbf4a0dc4901792053a3daf76968354dcb13bebd6325751b7e: Status 404 returned error can't find the container with id 8d66b2ae19d85dcbf4a0dc4901792053a3daf76968354dcb13bebd6325751b7e
Nov 02 18:05:00.727755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.727718    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"]
Nov 02 18:05:00.730666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.730629    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:00.730754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.730708    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:00.730754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:00.730741    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:05:01.034111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.034005    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" event={"ID":"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78","Type":"ContainerStarted","Data":"b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d"}
Nov 02 18:05:01.034111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.034076    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" event={"ID":"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78","Type":"ContainerStarted","Data":"8d66b2ae19d85dcbf4a0dc4901792053a3daf76968354dcb13bebd6325751b7e"}
Nov 02 18:05:01.035626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.035596    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk" event={"ID":"3977ce17-d1a7-4a34-a3a0-ddfb71a104ef","Type":"ContainerDied","Data":"a40f7139a2bc14e0744a32cfb20426b5e18314f20863b736752d29a71c2d174b"}
Nov 02 18:05:01.035747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.035651    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40f7139a2bc14e0744a32cfb20426b5e18314f20863b736752d29a71c2d174b"
Nov 02 18:05:01.035747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.035608    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"
Nov 02 18:05:01.066521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.066454    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podStartSLOduration=66.066434721 podStartE2EDuration="1m6.066434721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:01.064070275 +0000 UTC m=+1913.643479686" watchObservedRunningTime="2026-11-02 18:05:01.066434721 +0000 UTC m=+1913.645844109"
Nov 02 18:05:01.324736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.324646    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:01.326703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.326669    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:01.326860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.326734    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:01.546037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:01.546000    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038c55b6-2933-42e9-a931-3e9c9e6210c0" path="/var/lib/kubelet/pods/038c55b6-2933-42e9-a931-3e9c9e6210c0/volumes"
Nov 02 18:05:02.324131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:02.324097    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:02.324643 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:02.324160    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:03.325932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:03.325802    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:03.326455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:03.325956    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:04.324371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:04.324330    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:04.324602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:04.324393    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:05.329863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:05.327902    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:05.329863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:05.327965    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:06.324891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:06.324857    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:06.325092 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:06.324911    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:07.324410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:07.324369    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 18:05:07.325049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:07.324435    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 18:05:08.348862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:08.348494    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:08.353496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:08.350157    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:08.353496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:08.351173    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:05:18.678468 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:05:18.678269    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:05:35.082084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:35.082040    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"]
Nov 02 18:05:35.091250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:35.091206    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a454a017-t8hlx"]
Nov 02 18:05:35.546576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:35.546542    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb22fbd-150b-48ba-9e45-f4a1e792df8c" path="/var/lib/kubelet/pods/4eb22fbd-150b-48ba-9e45-f4a1e792df8c/volumes"
Nov 02 18:05:41.193920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:41.193883    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"]
Nov 02 18:05:45.042658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:45.042624    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"]
Nov 02 18:05:45.049375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:45.049340    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbccfafd-cgprf"]
Nov 02 18:05:45.546726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:45.546689    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc5a570-9741-4e52-b5fa-8e1b89e243a4" path="/var/lib/kubelet/pods/7dc5a570-9741-4e52-b5fa-8e1b89e243a4/volumes"
Nov 02 18:05:46.194736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.194663    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd" podUID="b114a0ef-16a0-4431-8831-bc28d99a6ff0" containerName="payload-processing" containerID="cri-o://bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033" gracePeriod=25
Nov 02 18:05:46.515094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.515062    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:05:46.557052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.557015    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"]
Nov 02 18:05:46.557934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.557911    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b114a0ef-16a0-4431-8831-bc28d99a6ff0" containerName="payload-processing"
Nov 02 18:05:46.557934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.557934    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b114a0ef-16a0-4431-8831-bc28d99a6ff0" containerName="payload-processing"
Nov 02 18:05:46.558128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.557980    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" containerName="revoke-keys"
Nov 02 18:05:46.558128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.557988    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" containerName="revoke-keys"
Nov 02 18:05:46.558249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.558229    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b114a0ef-16a0-4431-8831-bc28d99a6ff0" containerName="payload-processing"
Nov 02 18:05:46.558306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.558250    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" containerName="revoke-keys"
Nov 02 18:05:46.562091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.562069    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587065    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"kube-root-ca.crt\""
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587244    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"e2e-cmp-legacy-289d82-provider-cred\""
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587265    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"]
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587363    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-e2e--d2178d1f-dockercfg-rv9sj\""
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587418    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"payload-processing-external-model-plugins-e2e-extmod-a-807c08\""
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587490    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"openshift-service-ca.crt\""
Nov 02 18:05:46.588138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.587598    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-807c08\"/\"routing-overlay\""
Nov 02 18:05:46.596383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.596350    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume\") pod \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") "
Nov 02 18:05:46.597131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.597108    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume\") pod \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") "
Nov 02 18:05:46.598051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.598022    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials\") pod \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") "
Nov 02 18:05:46.598180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.596964    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume" pod "b114a0ef-16a0-4431-8831-bc28d99a6ff0" (UID: "b114a0ef-16a0-4431-8831-bc28d99a6ff0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:46.598303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.598283    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn\") pod \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\" (UID: \"b114a0ef-16a0-4431-8831-bc28d99a6ff0\") "
Nov 02 18:05:46.598490 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.598473    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume" pod "b114a0ef-16a0-4431-8831-bc28d99a6ff0" (UID: "b114a0ef-16a0-4431-8831-bc28d99a6ff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:46.599194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.599172    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:46.599299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.599199    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b114a0ef-16a0-4431-8831-bc28d99a6ff0-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:46.601331 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.601293    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials" pod "b114a0ef-16a0-4431-8831-bc28d99a6ff0" (UID: "b114a0ef-16a0-4431-8831-bc28d99a6ff0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:46.601880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.601809    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn" pod "b114a0ef-16a0-4431-8831-bc28d99a6ff0" (UID: "b114a0ef-16a0-4431-8831-bc28d99a6ff0"). InnerVolumeSpecName "kube-api-access-njfhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:46.740145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.736586    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.740145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.736637    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrq6\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.740145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.736761    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.740145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.736948    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.740145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.737044    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njfhn\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-kube-api-access-njfhn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:46.740145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.737060    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b114a0ef-16a0-4431-8831-bc28d99a6ff0-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:05:46.838320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.838283    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.838514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.838331    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrq6\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.838514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.838421    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.838650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.838603    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.839199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.839176    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.839455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.839430    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.841201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.841176    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.848467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.848442    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrq6\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6\") pod \"payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") " pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.898540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.898503    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:46.941703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.941620    5042 generic.go:356] "Generic (PLEG): container finished" podID="b114a0ef-16a0-4431-8831-bc28d99a6ff0" containerID="bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033" exitCode=0
Nov 02 18:05:46.941703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.941672    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"
Nov 02 18:05:46.941703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.941699    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd" event={"ID":"b114a0ef-16a0-4431-8831-bc28d99a6ff0","Type":"ContainerDied","Data":"bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033"}
Nov 02 18:05:46.942105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.941759    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd" event={"ID":"b114a0ef-16a0-4431-8831-bc28d99a6ff0","Type":"ContainerDied","Data":"d86b0561d26be3379bdfe1964f94b1ea48261be30cec0baf06ea0e250e283a7b"}
Nov 02 18:05:46.942105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.941784    5042 scope.go:122] "RemoveContainer" containerID="bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033"
Nov 02 18:05:46.956925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.956902    5042 scope.go:122] "RemoveContainer" containerID="bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033"
Nov 02 18:05:46.957225 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:05:46.957201    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033\": container with ID starting with bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033 not found: ID does not exist" containerID="bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033"
Nov 02 18:05:46.957293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.957234    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033"} err="failed to get container status \"bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033\": rpc error: code = NotFound desc = could not find container \"bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033\": container with ID starting with bdc4d1b707bb315ccc6e9fed58a2fe9467e3717e0b9d402033acc8da3def6033 not found: ID does not exist"
Nov 02 18:05:46.979572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.976289    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"]
Nov 02 18:05:46.986857 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:46.985122    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-1ef96f/payload-processing-external-model-e2e-extmod-b-1ef96f-7bd6fcsdd"]
Nov 02 18:05:47.196765 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:05:47.196726    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c36a1f_148b_4006_93e1_c3c1d0ddfda0.slice/crio-c53466f23a335b5d45bb72d4b0b532bab839d7890579ce3773afeef39c7a250e WatchSource:0}: Error finding container c53466f23a335b5d45bb72d4b0b532bab839d7890579ce3773afeef39c7a250e: Status 404 returned error can't find the container with id c53466f23a335b5d45bb72d4b0b532bab839d7890579ce3773afeef39c7a250e
Nov 02 18:05:47.199519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:47.199491    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"]
Nov 02 18:05:47.546272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:47.546194    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b114a0ef-16a0-4431-8831-bc28d99a6ff0" path="/var/lib/kubelet/pods/b114a0ef-16a0-4431-8831-bc28d99a6ff0/volumes"
Nov 02 18:05:47.952054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:47.952000    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv" event={"ID":"62c36a1f-148b-4006-93e1-c3c1d0ddfda0","Type":"ContainerStarted","Data":"c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792"}
Nov 02 18:05:47.952244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:47.952075    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:05:47.952244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:47.952096    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv" event={"ID":"62c36a1f-148b-4006-93e1-c3c1d0ddfda0","Type":"ContainerStarted","Data":"c53466f23a335b5d45bb72d4b0b532bab839d7890579ce3773afeef39c7a250e"}
Nov 02 18:05:47.975880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:47.975802    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv" podStartSLOduration=23.975788751 podStartE2EDuration="23.975788751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:47.9748164 +0000 UTC m=+1960.554225789" watchObservedRunningTime="2026-11-02 18:05:47.975788751 +0000 UTC m=+1960.555198137"
Nov 02 18:05:48.544018 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:05:48.543983    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:05:52.245878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:52.245689    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"]
Nov 02 18:05:52.292865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:52.283887    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"]
Nov 02 18:05:52.310867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:52.310741    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c2054d-ftbm9"]
Nov 02 18:05:52.354868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:52.352729    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220152a3-zjvqn"]
Nov 02 18:05:53.547371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:53.547332    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3e16960-7229-4ea9-a864-71f4449345cc" path="/var/lib/kubelet/pods/b3e16960-7229-4ea9-a864-71f4449345cc/volumes"
Nov 02 18:05:53.547914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:53.547898    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f9b794-4f22-4e6d-a851-b5d5dda79e38" path="/var/lib/kubelet/pods/d2f9b794-4f22-4e6d-a851-b5d5dda79e38/volumes"
Nov 02 18:05:56.030910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:56.030866    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"]
Nov 02 18:05:56.042585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:56.040323    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7ad00fc-26w47"]
Nov 02 18:05:57.548879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:57.548818    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582e7e73-ef9d-43ac-9717-f277deaf5ca5" path="/var/lib/kubelet/pods/582e7e73-ef9d-43ac-9717-f277deaf5ca5/volumes"
Nov 02 18:05:59.024573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:05:59.024536    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:06:02.031456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:02.031416    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"]
Nov 02 18:06:02.036550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:02.036522    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"]
Nov 02 18:06:02.041433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:02.041399    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6252674-hd4qr"]
Nov 02 18:06:02.045729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:02.045703    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d1da6fe-m4z6r"]
Nov 02 18:06:03.547411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:03.547375    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4803ff45-a5a9-4093-b1c9-a8244bcac5bb" path="/var/lib/kubelet/pods/4803ff45-a5a9-4093-b1c9-a8244bcac5bb/volumes"
Nov 02 18:06:03.547916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:03.547899    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1eb281-4071-4e0b-8b2f-43a7e59dee07" path="/var/lib/kubelet/pods/5b1eb281-4071-4e0b-8b2f-43a7e59dee07/volumes"
Nov 02 18:06:07.030865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:07.028230    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"]
Nov 02 18:06:07.067872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:07.062467    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5cf42b8-m59qd"]
Nov 02 18:06:07.547325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:07.547293    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d626b2f-9de2-47b7-be14-0e0ee402715a" path="/var/lib/kubelet/pods/5d626b2f-9de2-47b7-be14-0e0ee402715a/volumes"
Nov 02 18:06:18.543313 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:06:18.543280    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:06:24.992174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:24.992128    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"]
Nov 02 18:06:24.998503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:24.998479    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.009480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.007856    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:06:25.009480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.009198    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:06:25.012256 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.012225    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"]
Nov 02 18:06:25.097406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.097364    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr28h\" (UniqueName: \"kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.097895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.097868    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.097996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.097961    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.199454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.199417    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.199688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.199469    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.199688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.199528    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr28h\" (UniqueName: \"kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.200287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.200256    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.202136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.202107    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.209063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.209041    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr28h\" (UniqueName: \"kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h\") pod \"maas-api-revoke-sub-1bc60380-p4dqp\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.316657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.316565    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:25.676534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.676491    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"]
Nov 02 18:06:25.800738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:25.800699    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp" event={"ID":"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82","Type":"ContainerStarted","Data":"e4098634bc7d63d4f0c96406e56518f5ed2819ceaca58befd695cdc21f1bd06f"}
Nov 02 18:06:26.813222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:26.813180    5042 generic.go:356] "Generic (PLEG): container finished" podID="53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" containerID="a3b6af3577ffaadc76311f2edae9223be81b358c39e0042a29982e4bd3e1985e" exitCode=0
Nov 02 18:06:26.813720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:26.813263    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp" event={"ID":"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82","Type":"ContainerDied","Data":"a3b6af3577ffaadc76311f2edae9223be81b358c39e0042a29982e4bd3e1985e"}
Nov 02 18:06:28.028354 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.028324    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:28.128357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.128259    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h\" (UniqueName: \"kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h\") pod \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") "
Nov 02 18:06:28.128561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.128364    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca\") pod \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") "
Nov 02 18:06:28.128561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.128437    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca\") pod \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\" (UID: \"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82\") "
Nov 02 18:06:28.128815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.128789    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca" pod "53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" (UID: "53c8cbc5-4476-46f0-ba0a-b2c1fce20a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:06:28.129194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.129175    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:28.131267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.131246    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca" pod "53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" (UID: "53c8cbc5-4476-46f0-ba0a-b2c1fce20a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:06:28.131349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.131272    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h" pod "53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" (UID: "53c8cbc5-4476-46f0-ba0a-b2c1fce20a82"). InnerVolumeSpecName "kube-api-access-nr28h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:06:28.231340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.231303    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:28.231340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.231340    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr28h\" (UniqueName: \"kubernetes.io/projected/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82-kube-api-access-nr28h\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:28.837029 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.837000    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"
Nov 02 18:06:28.837260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.836997    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp" event={"ID":"53c8cbc5-4476-46f0-ba0a-b2c1fce20a82","Type":"ContainerDied","Data":"e4098634bc7d63d4f0c96406e56518f5ed2819ceaca58befd695cdc21f1bd06f"}
Nov 02 18:06:28.837260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:28.837143    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4098634bc7d63d4f0c96406e56518f5ed2819ceaca58befd695cdc21f1bd06f"
Nov 02 18:06:35.655738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.655702    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"]
Nov 02 18:06:35.656602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.656583    5042 cpu_manager.go:414] "containerMap: removing container" podUID="53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" containerName="revoke-keys"
Nov 02 18:06:35.656602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.656603    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" containerName="revoke-keys"
Nov 02 18:06:35.656783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.656774    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" containerName="revoke-keys"
Nov 02 18:06:35.660703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.660682    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.671348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.671319    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:06:35.672274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.672248    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:06:35.675504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.675479    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"]
Nov 02 18:06:35.788884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.788753    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.791633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.791605    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t65b\" (UniqueName: \"kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.791960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.791940    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.900718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.899536    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.900718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.899666    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t65b\" (UniqueName: \"kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.900718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.899745    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.907685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.907516    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.913910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.913828    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.916561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.916527    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t65b\" (UniqueName: \"kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b\") pod \"maas-api-revoke-sub-dbe3627d-k9pct\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:35.979310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:35.979269    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:36.285619 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:36.285594    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"]
Nov 02 18:06:36.932084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:36.932050    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct" event={"ID":"65937f44-72de-4086-a67d-65b8e01dd70d","Type":"ContainerStarted","Data":"bb1c86d18f4c75d475ea9dbae8cb42aa56171ea611a811a485d01ba47603a981"}
Nov 02 18:06:36.932462 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:36.932101    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct" event={"ID":"65937f44-72de-4086-a67d-65b8e01dd70d","Type":"ContainerStarted","Data":"05b75a1c4ceca9fb51324c66b886d8cad742012a4e7075bfcf5266bca73ffe55"}
Nov 02 18:06:36.953428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:36.953373    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct" podStartSLOduration=1.9533597729999999 podStartE2EDuration="1.953359773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:36.948689902 +0000 UTC m=+2009.528099302" watchObservedRunningTime="2026-11-02 18:06:36.953359773 +0000 UTC m=+2009.532769159"
Nov 02 18:06:37.942865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:37.942817    5042 generic.go:356] "Generic (PLEG): container finished" podID="65937f44-72de-4086-a67d-65b8e01dd70d" containerID="bb1c86d18f4c75d475ea9dbae8cb42aa56171ea611a811a485d01ba47603a981" exitCode=0
Nov 02 18:06:37.943297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:37.942911    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct" event={"ID":"65937f44-72de-4086-a67d-65b8e01dd70d","Type":"ContainerDied","Data":"bb1c86d18f4c75d475ea9dbae8cb42aa56171ea611a811a485d01ba47603a981"}
Nov 02 18:06:39.171675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.171649    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:39.336093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.336008    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca\") pod \"65937f44-72de-4086-a67d-65b8e01dd70d\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") "
Nov 02 18:06:39.336306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.336290    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca\") pod \"65937f44-72de-4086-a67d-65b8e01dd70d\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") "
Nov 02 18:06:39.336401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.336339    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b\" (UniqueName: \"kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b\") pod \"65937f44-72de-4086-a67d-65b8e01dd70d\" (UID: \"65937f44-72de-4086-a67d-65b8e01dd70d\") "
Nov 02 18:06:39.337637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.337584    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca" pod "65937f44-72de-4086-a67d-65b8e01dd70d" (UID: "65937f44-72de-4086-a67d-65b8e01dd70d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:06:39.339336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.339304    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca" pod "65937f44-72de-4086-a67d-65b8e01dd70d" (UID: "65937f44-72de-4086-a67d-65b8e01dd70d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:06:39.340269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.340240    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b" pod "65937f44-72de-4086-a67d-65b8e01dd70d" (UID: "65937f44-72de-4086-a67d-65b8e01dd70d"). InnerVolumeSpecName "kube-api-access-7t65b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:06:39.439067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.439035    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t65b\" (UniqueName: \"kubernetes.io/projected/65937f44-72de-4086-a67d-65b8e01dd70d-kube-api-access-7t65b\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:39.439067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.439067    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:39.439287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.439083    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65937f44-72de-4086-a67d-65b8e01dd70d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:39.962971 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.962938    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"
Nov 02 18:06:39.963170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.962971    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct" event={"ID":"65937f44-72de-4086-a67d-65b8e01dd70d","Type":"ContainerDied","Data":"05b75a1c4ceca9fb51324c66b886d8cad742012a4e7075bfcf5266bca73ffe55"}
Nov 02 18:06:39.963170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:39.963022    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05b75a1c4ceca9fb51324c66b886d8cad742012a4e7075bfcf5266bca73ffe55"
Nov 02 18:06:47.234179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.234144    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"]
Nov 02 18:06:47.234999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.234983    5042 cpu_manager.go:414] "containerMap: removing container" podUID="65937f44-72de-4086-a67d-65b8e01dd70d" containerName="revoke-keys"
Nov 02 18:06:47.235065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.235001    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65937f44-72de-4086-a67d-65b8e01dd70d" containerName="revoke-keys"
Nov 02 18:06:47.235170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.235153    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="65937f44-72de-4086-a67d-65b8e01dd70d" containerName="revoke-keys"
Nov 02 18:06:47.238919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.238901    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.247977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.247946    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"]
Nov 02 18:06:47.249636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.249612    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:06:47.249786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.249679    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:06:47.291620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.291573    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.291927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.291899    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsdr\" (UniqueName: \"kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.292240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.292216    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.394339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.394298    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsdr\" (UniqueName: \"kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.394529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.394513    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.394575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.394555    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.395439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.395344    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.397223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.397198    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.403248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.403216    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsdr\" (UniqueName: \"kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr\") pod \"maas-api-revoke-sub-948975e1-jdzvv\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.563569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.563477    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:47.912159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.912120    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"]
Nov 02 18:06:47.922240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.922206    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:47.935998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:47.935963    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"]
Nov 02 18:06:48.040752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.040716    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.041057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.041033    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5g6n\" (UniqueName: \"kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.041153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.041137    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.153938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.153299    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.153938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.153627    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5g6n\" (UniqueName: \"kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.153938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.153791    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.215754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.154679    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.280863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.268088    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5g6n\" (UniqueName: \"kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.286912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.286191    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fbef6f2-w8x9n\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.304683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.304401    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"]
Nov 02 18:06:48.315905 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:06:48.315816    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84daf268_b4be_482a_87cb_22e2b06055a7.slice/crio-f1056729e410853bc63d4210dfd6d1e35b8e06ded93d6002e110a9c4748a1815 WatchSource:0}: Error finding container f1056729e410853bc63d4210dfd6d1e35b8e06ded93d6002e110a9c4748a1815: Status 404 returned error can't find the container with id f1056729e410853bc63d4210dfd6d1e35b8e06ded93d6002e110a9c4748a1815
Nov 02 18:06:48.542475 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:06:48.542435    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:06:48.555012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.554981    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:48.916468 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:06:48.916423    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22631999_8bb9_470a_88a3_5cebb1123954.slice/crio-36dfdf0e44073e9a84d8fb9be21322c497591f2ae3424b4a6fc62a00a298699c WatchSource:0}: Error finding container 36dfdf0e44073e9a84d8fb9be21322c497591f2ae3424b4a6fc62a00a298699c: Status 404 returned error can't find the container with id 36dfdf0e44073e9a84d8fb9be21322c497591f2ae3424b4a6fc62a00a298699c
Nov 02 18:06:48.916827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:48.916806    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"]
Nov 02 18:06:49.263860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:49.263804    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv" event={"ID":"84daf268-b4be-482a-87cb-22e2b06055a7","Type":"ContainerStarted","Data":"bf9de72c8ce4fe83e594e270e869741d549b1ae1660e2c5537c011f2af690cfc"}
Nov 02 18:06:49.264437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:49.264348    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv" event={"ID":"84daf268-b4be-482a-87cb-22e2b06055a7","Type":"ContainerStarted","Data":"f1056729e410853bc63d4210dfd6d1e35b8e06ded93d6002e110a9c4748a1815"}
Nov 02 18:06:49.265889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:49.265823    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n" event={"ID":"22631999-8bb9-470a-88a3-5cebb1123954","Type":"ContainerStarted","Data":"36dfdf0e44073e9a84d8fb9be21322c497591f2ae3424b4a6fc62a00a298699c"}
Nov 02 18:06:49.283310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:49.283258    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv" podStartSLOduration=2.283242313 podStartE2EDuration="2.283242313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:49.281343442 +0000 UTC m=+2021.860752831" watchObservedRunningTime="2026-11-02 18:06:49.283242313 +0000 UTC m=+2021.862651701"
Nov 02 18:06:50.277561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:50.277528    5042 generic.go:356] "Generic (PLEG): container finished" podID="84daf268-b4be-482a-87cb-22e2b06055a7" containerID="bf9de72c8ce4fe83e594e270e869741d549b1ae1660e2c5537c011f2af690cfc" exitCode=0
Nov 02 18:06:50.277740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:50.277608    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv" event={"ID":"84daf268-b4be-482a-87cb-22e2b06055a7","Type":"ContainerDied","Data":"bf9de72c8ce4fe83e594e270e869741d549b1ae1660e2c5537c011f2af690cfc"}
Nov 02 18:06:50.279622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:50.279591    5042 generic.go:356] "Generic (PLEG): container finished" podID="22631999-8bb9-470a-88a3-5cebb1123954" containerID="6241a0b9eb6d553cf4b3ac9b29318b3384e09c1ccd15ae6214c054a4ea831339" exitCode=0
Nov 02 18:06:50.279736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:50.279679    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n" event={"ID":"22631999-8bb9-470a-88a3-5cebb1123954","Type":"ContainerDied","Data":"6241a0b9eb6d553cf4b3ac9b29318b3384e09c1ccd15ae6214c054a4ea831339"}
Nov 02 18:06:51.561492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.561095    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:51.568313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.568277    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:51.654518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.654477    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca\") pod \"84daf268-b4be-482a-87cb-22e2b06055a7\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") "
Nov 02 18:06:51.654710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.654620    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n\" (UniqueName: \"kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n\") pod \"22631999-8bb9-470a-88a3-5cebb1123954\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") "
Nov 02 18:06:51.654997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.654966    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca" pod "84daf268-b4be-482a-87cb-22e2b06055a7" (UID: "84daf268-b4be-482a-87cb-22e2b06055a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:06:51.655306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.655275    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca\") pod \"84daf268-b4be-482a-87cb-22e2b06055a7\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") "
Nov 02 18:06:51.655572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.655548    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr\" (UniqueName: \"kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr\") pod \"84daf268-b4be-482a-87cb-22e2b06055a7\" (UID: \"84daf268-b4be-482a-87cb-22e2b06055a7\") "
Nov 02 18:06:51.655715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.655638    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca\") pod \"22631999-8bb9-470a-88a3-5cebb1123954\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") "
Nov 02 18:06:51.655715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.655699    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca\") pod \"22631999-8bb9-470a-88a3-5cebb1123954\" (UID: \"22631999-8bb9-470a-88a3-5cebb1123954\") "
Nov 02 18:06:51.656237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.656193    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca" pod "22631999-8bb9-470a-88a3-5cebb1123954" (UID: "22631999-8bb9-470a-88a3-5cebb1123954"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:06:51.656987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.656796    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:51.656987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.656824    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:51.658108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.658079    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n" pod "22631999-8bb9-470a-88a3-5cebb1123954" (UID: "22631999-8bb9-470a-88a3-5cebb1123954"). InnerVolumeSpecName "kube-api-access-z5g6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:06:51.659556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.659498    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca" pod "84daf268-b4be-482a-87cb-22e2b06055a7" (UID: "84daf268-b4be-482a-87cb-22e2b06055a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:06:51.659744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.659710    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca" pod "22631999-8bb9-470a-88a3-5cebb1123954" (UID: "22631999-8bb9-470a-88a3-5cebb1123954"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:06:51.661009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.660978    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr" pod "84daf268-b4be-482a-87cb-22e2b06055a7" (UID: "84daf268-b4be-482a-87cb-22e2b06055a7"). InnerVolumeSpecName "kube-api-access-ntsdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:06:51.789943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.778155    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84daf268-b4be-482a-87cb-22e2b06055a7-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:51.796281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.791259    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsdr\" (UniqueName: \"kubernetes.io/projected/84daf268-b4be-482a-87cb-22e2b06055a7-kube-api-access-ntsdr\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:51.796281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.791324    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22631999-8bb9-470a-88a3-5cebb1123954-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:51.796281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:51.791338    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5g6n\" (UniqueName: \"kubernetes.io/projected/22631999-8bb9-470a-88a3-5cebb1123954-kube-api-access-z5g6n\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:52.309857 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:52.309817    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"
Nov 02 18:06:52.310082 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:52.309816    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv" event={"ID":"84daf268-b4be-482a-87cb-22e2b06055a7","Type":"ContainerDied","Data":"f1056729e410853bc63d4210dfd6d1e35b8e06ded93d6002e110a9c4748a1815"}
Nov 02 18:06:52.310082 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:52.309959    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1056729e410853bc63d4210dfd6d1e35b8e06ded93d6002e110a9c4748a1815"
Nov 02 18:06:52.311816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:52.311790    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"
Nov 02 18:06:52.311986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:52.311787    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n" event={"ID":"22631999-8bb9-470a-88a3-5cebb1123954","Type":"ContainerDied","Data":"36dfdf0e44073e9a84d8fb9be21322c497591f2ae3424b4a6fc62a00a298699c"}
Nov 02 18:06:52.311986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:52.311917    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36dfdf0e44073e9a84d8fb9be21322c497591f2ae3424b4a6fc62a00a298699c"
Nov 02 18:06:53.903068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.903026    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"]
Nov 02 18:06:53.904290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.904266    5042 cpu_manager.go:414] "containerMap: removing container" podUID="22631999-8bb9-470a-88a3-5cebb1123954" containerName="revoke-keys"
Nov 02 18:06:53.904359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.904294    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22631999-8bb9-470a-88a3-5cebb1123954" containerName="revoke-keys"
Nov 02 18:06:53.904359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.904346    5042 cpu_manager.go:414] "containerMap: removing container" podUID="84daf268-b4be-482a-87cb-22e2b06055a7" containerName="revoke-keys"
Nov 02 18:06:53.904359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.904355    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84daf268-b4be-482a-87cb-22e2b06055a7" containerName="revoke-keys"
Nov 02 18:06:53.904589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.904576    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="22631999-8bb9-470a-88a3-5cebb1123954" containerName="revoke-keys"
Nov 02 18:06:53.905747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.904598    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="84daf268-b4be-482a-87cb-22e2b06055a7" containerName="revoke-keys"
Nov 02 18:06:53.927802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.926450    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"]
Nov 02 18:06:53.927802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.926584    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:53.931638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.931615    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:06:53.931789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:53.931638    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:06:54.040372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.040331    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.040586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.040529    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.040704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.040687    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dnw\" (UniqueName: \"kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.146862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.146094    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.146862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.146354    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dnw\" (UniqueName: \"kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.146862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.146468    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.153502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.151884    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.171921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.158334    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.171921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.160735    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dnw\" (UniqueName: \"kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw\") pod \"maas-api-revoke-sub-373e7f75-w7hkp\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.284671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.284636    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:54.610211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:54.596588    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"]
Nov 02 18:06:54.610211 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:06:54.597607    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc7f92f_59bf_4176_991b_2751eb5df590.slice/crio-9f5bce165c169d50828f4510f81e2d7b6dc5f96700f83a26f065c022f408813b WatchSource:0}: Error finding container 9f5bce165c169d50828f4510f81e2d7b6dc5f96700f83a26f065c022f408813b: Status 404 returned error can't find the container with id 9f5bce165c169d50828f4510f81e2d7b6dc5f96700f83a26f065c022f408813b
Nov 02 18:06:55.085943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.072829    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"]
Nov 02 18:06:55.092289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.092253    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"]
Nov 02 18:06:55.092413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.092373    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.173159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.172366    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.173159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.172518    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.173159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.172766    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2m8\" (UniqueName: \"kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.274827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.274787    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.275069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.274894    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.275186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.275151    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2m8\" (UniqueName: \"kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.275906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.275880    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.277523 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.277501    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.285005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.284976    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2m8\" (UniqueName: \"kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8\") pod \"maas-api-revoke-sub-e1694dc9-cwl6s\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.351386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.351348    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp" event={"ID":"dfc7f92f-59bf-4176-991b-2751eb5df590","Type":"ContainerStarted","Data":"e58887657b4702de298ab3d6b59b765de516cffe2a6e86ad59c3e21c9de8dcbf"}
Nov 02 18:06:55.351567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.351408    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp" event={"ID":"dfc7f92f-59bf-4176-991b-2751eb5df590","Type":"ContainerStarted","Data":"9f5bce165c169d50828f4510f81e2d7b6dc5f96700f83a26f065c022f408813b"}
Nov 02 18:06:55.376400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.376346    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp" podStartSLOduration=2.376331065 podStartE2EDuration="2.376331065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:55.368801432 +0000 UTC m=+2027.948210819" watchObservedRunningTime="2026-11-02 18:06:55.376331065 +0000 UTC m=+2027.955740451"
Nov 02 18:06:55.416897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.416861    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:55.750536 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:06:55.750507    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805d215c_170f_4e5c_8500_b1a0d5d069a2.slice/crio-c86dcdf24b3641df1bcec6cb50f6c82597dbfecf1f152b3e4cd7d7a9ccb3da85 WatchSource:0}: Error finding container c86dcdf24b3641df1bcec6cb50f6c82597dbfecf1f152b3e4cd7d7a9ccb3da85: Status 404 returned error can't find the container with id c86dcdf24b3641df1bcec6cb50f6c82597dbfecf1f152b3e4cd7d7a9ccb3da85
Nov 02 18:06:55.750828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:55.750805    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"]
Nov 02 18:06:56.362111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:56.362079    5042 generic.go:356] "Generic (PLEG): container finished" podID="dfc7f92f-59bf-4176-991b-2751eb5df590" containerID="e58887657b4702de298ab3d6b59b765de516cffe2a6e86ad59c3e21c9de8dcbf" exitCode=0
Nov 02 18:06:56.362583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:56.362162    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp" event={"ID":"dfc7f92f-59bf-4176-991b-2751eb5df590","Type":"ContainerDied","Data":"e58887657b4702de298ab3d6b59b765de516cffe2a6e86ad59c3e21c9de8dcbf"}
Nov 02 18:06:56.364206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:56.364177    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s" event={"ID":"805d215c-170f-4e5c-8500-b1a0d5d069a2","Type":"ContainerStarted","Data":"8ae9f7329d7b0570d1887178ff3799d9cc5e30b7eb1a2b064a05b74d4feb7b87"}
Nov 02 18:06:56.364324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:56.364228    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s" event={"ID":"805d215c-170f-4e5c-8500-b1a0d5d069a2","Type":"ContainerStarted","Data":"c86dcdf24b3641df1bcec6cb50f6c82597dbfecf1f152b3e4cd7d7a9ccb3da85"}
Nov 02 18:06:56.408791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:56.408729    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s" podStartSLOduration=1.408710226 podStartE2EDuration="1.408710226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:56.401376621 +0000 UTC m=+2028.980786008" watchObservedRunningTime="2026-11-02 18:06:56.408710226 +0000 UTC m=+2028.988119614"
Nov 02 18:06:57.378392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.378353    5042 generic.go:356] "Generic (PLEG): container finished" podID="805d215c-170f-4e5c-8500-b1a0d5d069a2" containerID="8ae9f7329d7b0570d1887178ff3799d9cc5e30b7eb1a2b064a05b74d4feb7b87" exitCode=0
Nov 02 18:06:57.378913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.378437    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s" event={"ID":"805d215c-170f-4e5c-8500-b1a0d5d069a2","Type":"ContainerDied","Data":"8ae9f7329d7b0570d1887178ff3799d9cc5e30b7eb1a2b064a05b74d4feb7b87"}
Nov 02 18:06:57.636688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.636616    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:57.744061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.743534    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw\" (UniqueName: \"kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw\") pod \"dfc7f92f-59bf-4176-991b-2751eb5df590\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") "
Nov 02 18:06:57.744061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.743719    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca\") pod \"dfc7f92f-59bf-4176-991b-2751eb5df590\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") "
Nov 02 18:06:57.744061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.743750    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca\") pod \"dfc7f92f-59bf-4176-991b-2751eb5df590\" (UID: \"dfc7f92f-59bf-4176-991b-2751eb5df590\") "
Nov 02 18:06:57.744423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.744187    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca" pod "dfc7f92f-59bf-4176-991b-2751eb5df590" (UID: "dfc7f92f-59bf-4176-991b-2751eb5df590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:06:57.744504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.744484    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:57.746425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.746388    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca" pod "dfc7f92f-59bf-4176-991b-2751eb5df590" (UID: "dfc7f92f-59bf-4176-991b-2751eb5df590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:06:57.746638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.746620    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw" pod "dfc7f92f-59bf-4176-991b-2751eb5df590" (UID: "dfc7f92f-59bf-4176-991b-2751eb5df590"). InnerVolumeSpecName "kube-api-access-g5dnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:06:57.861060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.852893    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc7f92f-59bf-4176-991b-2751eb5df590-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:57.861060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:57.852944    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5dnw\" (UniqueName: \"kubernetes.io/projected/dfc7f92f-59bf-4176-991b-2751eb5df590-kube-api-access-g5dnw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:58.098394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:58.098352    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"]
Nov 02 18:06:58.462909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:58.459624    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"
Nov 02 18:06:58.470920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:58.469938    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp" event={"ID":"dfc7f92f-59bf-4176-991b-2751eb5df590","Type":"ContainerDied","Data":"9f5bce165c169d50828f4510f81e2d7b6dc5f96700f83a26f065c022f408813b"}
Nov 02 18:06:58.470920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:58.470015    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f5bce165c169d50828f4510f81e2d7b6dc5f96700f83a26f065c022f408813b"
Nov 02 18:06:59.219900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.218628    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:06:59.407923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.394097    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca\") pod \"805d215c-170f-4e5c-8500-b1a0d5d069a2\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") "
Nov 02 18:06:59.407923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.394188    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8\" (UniqueName: \"kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8\") pod \"805d215c-170f-4e5c-8500-b1a0d5d069a2\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") "
Nov 02 18:06:59.407923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.394347    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca\") pod \"805d215c-170f-4e5c-8500-b1a0d5d069a2\" (UID: \"805d215c-170f-4e5c-8500-b1a0d5d069a2\") "
Nov 02 18:06:59.431954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.416042    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca" pod "805d215c-170f-4e5c-8500-b1a0d5d069a2" (UID: "805d215c-170f-4e5c-8500-b1a0d5d069a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:06:59.640908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.503615    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:59.748932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.748892    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca" pod "805d215c-170f-4e5c-8500-b1a0d5d069a2" (UID: "805d215c-170f-4e5c-8500-b1a0d5d069a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:06:59.771244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.758357    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8" pod "805d215c-170f-4e5c-8500-b1a0d5d069a2" (UID: "805d215c-170f-4e5c-8500-b1a0d5d069a2"). InnerVolumeSpecName "kube-api-access-zt2m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:06:59.877493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.876440    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/805d215c-170f-4e5c-8500-b1a0d5d069a2-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:06:59.877493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:06:59.876477    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2m8\" (UniqueName: \"kubernetes.io/projected/805d215c-170f-4e5c-8500-b1a0d5d069a2-kube-api-access-zt2m8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:00.114727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:00.114630    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s" event={"ID":"805d215c-170f-4e5c-8500-b1a0d5d069a2","Type":"ContainerDied","Data":"c86dcdf24b3641df1bcec6cb50f6c82597dbfecf1f152b3e4cd7d7a9ccb3da85"}
Nov 02 18:07:00.115049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:00.115013    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c86dcdf24b3641df1bcec6cb50f6c82597dbfecf1f152b3e4cd7d7a9ccb3da85"
Nov 02 18:07:00.115474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:00.115443    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"
Nov 02 18:07:03.099037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.098955    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv" podUID="62c36a1f-148b-4006-93e1-c3c1d0ddfda0" containerName="payload-processing" containerID="cri-o://c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792" gracePeriod=25
Nov 02 18:07:03.407738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.407702    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:07:03.510189 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.510155    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume\") pod \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") "
Nov 02 18:07:03.510644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.510209    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume\") pod \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") "
Nov 02 18:07:03.510644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.510281    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials\") pod \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") "
Nov 02 18:07:03.510793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.510650    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6\") pod \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\" (UID: \"62c36a1f-148b-4006-93e1-c3c1d0ddfda0\") "
Nov 02 18:07:03.510793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.510649    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume" pod "62c36a1f-148b-4006-93e1-c3c1d0ddfda0" (UID: "62c36a1f-148b-4006-93e1-c3c1d0ddfda0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:03.510793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.510675    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume" pod "62c36a1f-148b-4006-93e1-c3c1d0ddfda0" (UID: "62c36a1f-148b-4006-93e1-c3c1d0ddfda0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:03.511359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.511337    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:03.511359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.511358    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:03.513292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.513260    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6" pod "62c36a1f-148b-4006-93e1-c3c1d0ddfda0" (UID: "62c36a1f-148b-4006-93e1-c3c1d0ddfda0"). InnerVolumeSpecName "kube-api-access-dbrq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:03.513409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.513384    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials" pod "62c36a1f-148b-4006-93e1-c3c1d0ddfda0" (UID: "62c36a1f-148b-4006-93e1-c3c1d0ddfda0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:03.542614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.542580    5042 generic.go:356] "Generic (PLEG): container finished" podID="62c36a1f-148b-4006-93e1-c3c1d0ddfda0" containerID="c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792" exitCode=0
Nov 02 18:07:03.542778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.542707    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"
Nov 02 18:07:03.547465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.547429    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv" event={"ID":"62c36a1f-148b-4006-93e1-c3c1d0ddfda0","Type":"ContainerDied","Data":"c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792"}
Nov 02 18:07:03.547604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.547478    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv" event={"ID":"62c36a1f-148b-4006-93e1-c3c1d0ddfda0","Type":"ContainerDied","Data":"c53466f23a335b5d45bb72d4b0b532bab839d7890579ce3773afeef39c7a250e"}
Nov 02 18:07:03.547604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.547501    5042 scope.go:122] "RemoveContainer" containerID="c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792"
Nov 02 18:07:03.562116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.562091    5042 scope.go:122] "RemoveContainer" containerID="c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792"
Nov 02 18:07:03.562387 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:03.562367    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792\": container with ID starting with c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792 not found: ID does not exist" containerID="c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792"
Nov 02 18:07:03.562429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.562395    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792"} err="failed to get container status \"c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792\": rpc error: code = NotFound desc = could not find container \"c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792\": container with ID starting with c24170c764e22f36a42b6410698f35c0b25bba41c545e4ea6d1d67585dfcb792 not found: ID does not exist"
Nov 02 18:07:03.575435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.575405    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"]
Nov 02 18:07:03.580480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.580453    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-807c08/payload-processing-external-model-e2e-extmod-a-807c08-7b66gwdtv"]
Nov 02 18:07:03.649919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.618744    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:03.650285 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:03.650257    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrq6\" (UniqueName: \"kubernetes.io/projected/62c36a1f-148b-4006-93e1-c3c1d0ddfda0-kube-api-access-dbrq6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:05.742861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:05.737776    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c36a1f-148b-4006-93e1-c3c1d0ddfda0" path="/var/lib/kubelet/pods/62c36a1f-148b-4006-93e1-c3c1d0ddfda0/volumes"
Nov 02 18:07:07.355792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.353820    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"]
Nov 02 18:07:07.356336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356127    5042 cpu_manager.go:414] "containerMap: removing container" podUID="62c36a1f-148b-4006-93e1-c3c1d0ddfda0" containerName="payload-processing"
Nov 02 18:07:07.356336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356180    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c36a1f-148b-4006-93e1-c3c1d0ddfda0" containerName="payload-processing"
Nov 02 18:07:07.356336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356214    5042 cpu_manager.go:414] "containerMap: removing container" podUID="805d215c-170f-4e5c-8500-b1a0d5d069a2" containerName="revoke-keys"
Nov 02 18:07:07.356336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356224    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805d215c-170f-4e5c-8500-b1a0d5d069a2" containerName="revoke-keys"
Nov 02 18:07:07.356336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356303    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dfc7f92f-59bf-4176-991b-2751eb5df590" containerName="revoke-keys"
Nov 02 18:07:07.356336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356334    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc7f92f-59bf-4176-991b-2751eb5df590" containerName="revoke-keys"
Nov 02 18:07:07.356700 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356671    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="805d215c-170f-4e5c-8500-b1a0d5d069a2" containerName="revoke-keys"
Nov 02 18:07:07.356759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356724    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc7f92f-59bf-4176-991b-2751eb5df590" containerName="revoke-keys"
Nov 02 18:07:07.356759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.356751    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c36a1f-148b-4006-93e1-c3c1d0ddfda0" containerName="payload-processing"
Nov 02 18:07:07.369228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.363913    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.375283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.375157    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:07:07.381744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.378020    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"]
Nov 02 18:07:07.418030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.417989    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8h4\" (UniqueName: \"kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.418278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.418242    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.418528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.418509    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.440420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.440384    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:07:07.459613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.459577    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"]
Nov 02 18:07:07.465615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.465578    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.474159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.474128    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"agc-cred-c7aa0fd5ea16\""
Nov 02 18:07:07.474327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.474295    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-e2e--a4661749-dockercfg-vq976\""
Nov 02 18:07:07.474942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.474714    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"]
Nov 02 18:07:07.479911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.478952    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"kube-root-ca.crt\""
Nov 02 18:07:07.479911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.478971    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"routing-overlay\""
Nov 02 18:07:07.479911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.478957    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"openshift-service-ca.crt\""
Nov 02 18:07:07.479911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.479015    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a8e574\""
Nov 02 18:07:07.521068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521027    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x277k\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.521291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521244    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.521371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521346    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.521502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521475    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8h4\" (UniqueName: \"kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.521706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521684    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.521803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521731    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.521929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.521907    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.523053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.523017    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.526886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.526817    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.532854 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.532805    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8h4\" (UniqueName: \"kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4\") pod \"maas-api-revoke-sub-9cc40b61-4thlg\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.666922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.659593    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.666922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.659750    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.712869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.711827    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.712869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.712179    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x277k\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.712869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.712362    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.712869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.712566    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.750867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.739398    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"kube-root-ca.crt\""
Nov 02 18:07:07.776876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.760265    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"openshift-service-ca.crt\""
Nov 02 18:07:07.776876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.761050    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"agc-cred-c7aa0fd5ea16\""
Nov 02 18:07:07.790883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.790224    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:07:07.790883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.790286    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:07.859892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.842420    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:07.860934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:07.860799    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x277k\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k\") pod \"payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") " pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:08.123413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:08.122471    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a8e574\"/\"payload-processing-external-model-e2e--a4661749-dockercfg-vq976\""
Nov 02 18:07:08.123413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:08.122565    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:08.512938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:08.512908    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"]
Nov 02 18:07:08.513442 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:07:08.513412    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b86aa3_38d8_4a9c_8892_2dd81a992c29.slice/crio-641f7e66c7e295b7e4c9474c9f916d39fec08d8ec7228e426db9b7eff1411e65 WatchSource:0}: Error finding container 641f7e66c7e295b7e4c9474c9f916d39fec08d8ec7228e426db9b7eff1411e65: Status 404 returned error can't find the container with id 641f7e66c7e295b7e4c9474c9f916d39fec08d8ec7228e426db9b7eff1411e65
Nov 02 18:07:08.577116 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:07:08.577079    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277e3f59_79bb_4760_9513_b8c1583ba28a.slice/crio-064961845d7a4dd319f5c15d894a35d23d2cbe8629c7274963fad21ed7a7facb WatchSource:0}: Error finding container 064961845d7a4dd319f5c15d894a35d23d2cbe8629c7274963fad21ed7a7facb: Status 404 returned error can't find the container with id 064961845d7a4dd319f5c15d894a35d23d2cbe8629c7274963fad21ed7a7facb
Nov 02 18:07:08.577246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:08.577192    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"]
Nov 02 18:07:09.155943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:09.155907    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg" event={"ID":"e5b86aa3-38d8-4a9c-8892-2dd81a992c29","Type":"ContainerStarted","Data":"641f7e66c7e295b7e4c9474c9f916d39fec08d8ec7228e426db9b7eff1411e65"}
Nov 02 18:07:09.158490 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:09.158461    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" event={"ID":"277e3f59-79bb-4760-9513-b8c1583ba28a","Type":"ContainerStarted","Data":"07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9"}
Nov 02 18:07:09.158653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:09.158519    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" event={"ID":"277e3f59-79bb-4760-9513-b8c1583ba28a","Type":"ContainerStarted","Data":"064961845d7a4dd319f5c15d894a35d23d2cbe8629c7274963fad21ed7a7facb"}
Nov 02 18:07:09.158653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:09.158562    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:09.190186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:09.190124    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg" podStartSLOduration=2.190104693 podStartE2EDuration="2.190104693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:09.17273169 +0000 UTC m=+2041.752141100" watchObservedRunningTime="2026-11-02 18:07:09.190104693 +0000 UTC m=+2041.769514079"
Nov 02 18:07:09.202575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:09.202506    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" podStartSLOduration=2.202479593 podStartE2EDuration="2.202479593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:09.194463326 +0000 UTC m=+2041.773872712" watchObservedRunningTime="2026-11-02 18:07:09.202479593 +0000 UTC m=+2041.781888982"
Nov 02 18:07:10.408788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:10.408747    5042 generic.go:356] "Generic (PLEG): container finished" podID="e5b86aa3-38d8-4a9c-8892-2dd81a992c29" containerID="ca5f48f108b6f0915fc58a5ba8a77bc89027373811d2b22c6833f1b7ab054e64" exitCode=0
Nov 02 18:07:10.409286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:10.408903    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg" event={"ID":"e5b86aa3-38d8-4a9c-8892-2dd81a992c29","Type":"ContainerDied","Data":"ca5f48f108b6f0915fc58a5ba8a77bc89027373811d2b22c6833f1b7ab054e64"}
Nov 02 18:07:10.537192 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:10.537102    5042 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:10.537192 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:10.537131    5042 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:10.537433 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:10.537203    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials podName:277e3f59-79bb-4760-9513-b8c1583ba28a nodeName:}" failed. No retries permitted until 2026-11-02 18:07:11.037177239 +0000 UTC m=+2043.616586606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a") : secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:11.105182 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:11.077321    5042 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:11.106918 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:11.106893    5042 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:11.107195 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:11.107181    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials podName:277e3f59-79bb-4760-9513-b8c1583ba28a nodeName:}" failed. No retries permitted until 2026-11-02 18:07:12.107155908 +0000 UTC m=+2044.686565294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a") : secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:11.611484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.611455    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:11.687671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.687579    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca\") pod \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") "
Nov 02 18:07:11.687885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.687704    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca\") pod \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") "
Nov 02 18:07:11.687885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.687852    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4\" (UniqueName: \"kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4\") pod \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\" (UID: \"e5b86aa3-38d8-4a9c-8892-2dd81a992c29\") "
Nov 02 18:07:11.688022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.687994    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca" pod "e5b86aa3-38d8-4a9c-8892-2dd81a992c29" (UID: "e5b86aa3-38d8-4a9c-8892-2dd81a992c29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:11.688997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.688971    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:11.690344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.690311    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4" pod "e5b86aa3-38d8-4a9c-8892-2dd81a992c29" (UID: "e5b86aa3-38d8-4a9c-8892-2dd81a992c29"). InnerVolumeSpecName "kube-api-access-2q8h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:11.690344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.690337    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca" pod "e5b86aa3-38d8-4a9c-8892-2dd81a992c29" (UID: "e5b86aa3-38d8-4a9c-8892-2dd81a992c29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:07:11.796755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.796713    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:11.797035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:11.797017    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8h4\" (UniqueName: \"kubernetes.io/projected/e5b86aa3-38d8-4a9c-8892-2dd81a992c29-kube-api-access-2q8h4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:12.204521 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:12.204474    5042 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:12.204521 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:12.204513    5042 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:12.204803 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:12.204582    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials podName:277e3f59-79bb-4760-9513-b8c1583ba28a nodeName:}" failed. No retries permitted until 2026-11-02 18:07:14.204563987 +0000 UTC m=+2046.783973353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a") : secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:12.531706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:12.531616    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg" event={"ID":"e5b86aa3-38d8-4a9c-8892-2dd81a992c29","Type":"ContainerDied","Data":"641f7e66c7e295b7e4c9474c9f916d39fec08d8ec7228e426db9b7eff1411e65"}
Nov 02 18:07:12.531976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:12.531958    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641f7e66c7e295b7e4c9474c9f916d39fec08d8ec7228e426db9b7eff1411e65"
Nov 02 18:07:12.532209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:12.532129    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"
Nov 02 18:07:14.237538 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:14.237496    5042 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:14.237538 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:14.237536    5042 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:14.238125 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:14.237622    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials podName:277e3f59-79bb-4760-9513-b8c1583ba28a nodeName:}" failed. No retries permitted until 2026-11-02 18:07:18.237601073 +0000 UTC m=+2050.817010449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a") : secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:14.697937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.697896    5042 scope.go:122] "RemoveContainer" containerID="d98c8107171090e580cfd9b05a5584b5a2d1d9b410178218afa70d13a7111fd1"
Nov 02 18:07:14.728408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.728384    5042 scope.go:122] "RemoveContainer" containerID="191a7f66245422bc42cc8c83d983859752a1b924a81eb491f6c3d13077a58ed6"
Nov 02 18:07:14.749491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.749455    5042 scope.go:122] "RemoveContainer" containerID="377849398f98709203f30eac2ddc849a2d548d8daab9f583ef216d5426ef0ae1"
Nov 02 18:07:14.761636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.761611    5042 scope.go:122] "RemoveContainer" containerID="0e097a678aee6a309ce38e00a4dc07c551d0e1e08e58f6db8d3c9050df36106e"
Nov 02 18:07:14.773794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.773768    5042 scope.go:122] "RemoveContainer" containerID="78a10e1d5bfecf95c912ee2bbb7b7392cbbf3c9e7b7dbb32af6b19ad6fdf6f21"
Nov 02 18:07:14.797237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.797214    5042 scope.go:122] "RemoveContainer" containerID="8dbbffea7337a784f0a72bb23a370126089f5eb04a0b10c7c2b46c5967e6ca2e"
Nov 02 18:07:14.809914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.809889    5042 scope.go:122] "RemoveContainer" containerID="5b0e498c0f911f24950e5d552ad080fd376aa1cbe3eee392e39eacbf2979b648"
Nov 02 18:07:14.823538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:14.823514    5042 scope.go:122] "RemoveContainer" containerID="2872f80fc7350661ac30eb5eb900642a702b9c09606e7e0ee65ff918f842dd54"
Nov 02 18:07:17.007604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:17.007563    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"]
Nov 02 18:07:17.008476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:17.008440    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy" containerID="cri-o://a198bf285cc421c8ac3777debc03677a452dc1acdb504fffe4cdd461b85cc32d" gracePeriod=30
Nov 02 18:07:18.293966 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:18.293911    5042 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:18.293966 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:18.293962    5042 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:18.294810 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:18.294046    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials podName:277e3f59-79bb-4760-9513-b8c1583ba28a nodeName:}" failed. No retries permitted until 2026-11-02 18:07:26.294026448 +0000 UTC m=+2058.873435826 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a") : secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:18.541968 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:18.541930    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:07:19.531109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.531077    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"]
Nov 02 18:07:19.531962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.531944    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e5b86aa3-38d8-4a9c-8892-2dd81a992c29" containerName="revoke-keys"
Nov 02 18:07:19.531962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.531962    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b86aa3-38d8-4a9c-8892-2dd81a992c29" containerName="revoke-keys"
Nov 02 18:07:19.532101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.532092    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b86aa3-38d8-4a9c-8892-2dd81a992c29" containerName="revoke-keys"
Nov 02 18:07:19.539262 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.538278    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.543044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.543015    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:07:19.543919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.543879    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:07:19.557772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.555300    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"]
Nov 02 18:07:19.629706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.629669    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.630020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.630002    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nptww\" (UniqueName: \"kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.630343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.630321    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.732575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.732532    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nptww\" (UniqueName: \"kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.732807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.732788    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.732895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.732865    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.733470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.733450    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.735531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.735508    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.741611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.741591    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nptww\" (UniqueName: \"kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww\") pod \"maas-api-revoke-sub-8e396bc0-64rgg\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:19.869760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:19.869675    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:20.171018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:20.170989    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"]
Nov 02 18:07:20.171518 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:07:20.171490    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b08b8bd_23cf_4ac8_96c6_e0e65d38a225.slice/crio-e0a3b800552cfa01f48a83a214baa2e0996cd5c15457013916ca3ca32c2dc9d3 WatchSource:0}: Error finding container e0a3b800552cfa01f48a83a214baa2e0996cd5c15457013916ca3ca32c2dc9d3: Status 404 returned error can't find the container with id e0a3b800552cfa01f48a83a214baa2e0996cd5c15457013916ca3ca32c2dc9d3
Nov 02 18:07:20.413103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:20.413074    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:20.657332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:20.657297    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg" event={"ID":"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225","Type":"ContainerStarted","Data":"04df5d9667d9cca6b8d4ec537deb8cb61fd246de06a6b4fa91c7e9773694f0db"}
Nov 02 18:07:20.657742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:20.657349    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg" event={"ID":"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225","Type":"ContainerStarted","Data":"e0a3b800552cfa01f48a83a214baa2e0996cd5c15457013916ca3ca32c2dc9d3"}
Nov 02 18:07:20.680488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:20.680417    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg" podStartSLOduration=1.6803978960000001 podStartE2EDuration="1.680397896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:20.673724724 +0000 UTC m=+2053.253134111" watchObservedRunningTime="2026-11-02 18:07:20.680397896 +0000 UTC m=+2053.259807284"
Nov 02 18:07:21.667637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:21.667541    5042 generic.go:356] "Generic (PLEG): container finished" podID="2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" containerID="04df5d9667d9cca6b8d4ec537deb8cb61fd246de06a6b4fa91c7e9773694f0db" exitCode=0
Nov 02 18:07:21.667637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:21.667582    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg" event={"ID":"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225","Type":"ContainerDied","Data":"04df5d9667d9cca6b8d4ec537deb8cb61fd246de06a6b4fa91c7e9773694f0db"}
Nov 02 18:07:22.451633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:22.451591    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"]
Nov 02 18:07:22.850934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:22.848534    5042 generic.go:356] "Generic (PLEG): container finished" podID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerID="a198bf285cc421c8ac3777debc03677a452dc1acdb504fffe4cdd461b85cc32d" exitCode=0
Nov 02 18:07:22.850934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:22.848591    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" event={"ID":"d7d700c8-9e21-4d93-964f-aad473eb5dcd","Type":"ContainerDied","Data":"a198bf285cc421c8ac3777debc03677a452dc1acdb504fffe4cdd461b85cc32d"}
Nov 02 18:07:23.539609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.539577    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 18:07:23.574763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.574735    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.575490 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.575465    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.575761 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.575742    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.575963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.575946    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.576247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.576230    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.576611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.575753    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:23.576689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.575993    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:23.576689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.576495    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:23.576810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.576794    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.576959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.576889    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.577024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.576956    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.577024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.576994    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert\") pod \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\" (UID: \"d7d700c8-9e21-4d93-964f-aad473eb5dcd\") "
Nov 02 18:07:23.578280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.577821    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:23.580978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.580735    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.580978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.580967    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.581156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.580991    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.581156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.581005    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.581720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.581701    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:23.581880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.581720    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "kube-api-access-9hxjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:23.585760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.585735    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:23.588640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.588608    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:23.589209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.589189    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:23.589351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.589332    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo" pod "d7d700c8-9e21-4d93-964f-aad473eb5dcd" (UID: "d7d700c8-9e21-4d93-964f-aad473eb5dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:07:23.688102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.687634    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca\") pod \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") "
Nov 02 18:07:23.688102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.687753    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca\") pod \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") "
Nov 02 18:07:23.688102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.687867    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww\" (UniqueName: \"kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww\") pod \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\" (UID: \"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225\") "
Nov 02 18:07:23.691197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.689019    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.691197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.689050    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.691197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.689065    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hxjl\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-kube-api-access-9hxjl\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.691197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.689080    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.691197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.689096    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7d700c8-9e21-4d93-964f-aad473eb5dcd-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.697505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.692498    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww" pod "2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" (UID: "2b08b8bd-23cf-4ac8-96c6-e0e65d38a225"). InnerVolumeSpecName "kube-api-access-nptww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:23.697505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.696675    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca" pod "2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" (UID: "2b08b8bd-23cf-4ac8-96c6-e0e65d38a225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:07:23.697505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.696894    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca" pod "2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" (UID: "2b08b8bd-23cf-4ac8-96c6-e0e65d38a225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:23.790468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.790375    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.790468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.790403    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.790468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.790413    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nptww\" (UniqueName: \"kubernetes.io/projected/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225-kube-api-access-nptww\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:23.880230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.880172    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"
Nov 02 18:07:23.880727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.880169    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b" event={"ID":"d7d700c8-9e21-4d93-964f-aad473eb5dcd","Type":"ContainerDied","Data":"45f195dde07fb0ffb54287cf4902f3248df272fbdbb556cbb5336ca04d3aa4e0"}
Nov 02 18:07:23.880789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.880751    5042 scope.go:122] "RemoveContainer" containerID="a198bf285cc421c8ac3777debc03677a452dc1acdb504fffe4cdd461b85cc32d"
Nov 02 18:07:23.888128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.888101    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg" event={"ID":"2b08b8bd-23cf-4ac8-96c6-e0e65d38a225","Type":"ContainerDied","Data":"e0a3b800552cfa01f48a83a214baa2e0996cd5c15457013916ca3ca32c2dc9d3"}
Nov 02 18:07:23.888211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.888162    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0a3b800552cfa01f48a83a214baa2e0996cd5c15457013916ca3ca32c2dc9d3"
Nov 02 18:07:23.888211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.888112    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"
Nov 02 18:07:23.919138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.919087    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"]
Nov 02 18:07:23.928877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:23.928811    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-jzt5b"]
Nov 02 18:07:24.449865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:24.449335    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"]
Nov 02 18:07:24.449865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:24.449626    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy" containerID="cri-o://7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b" gracePeriod=30
Nov 02 18:07:25.548558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:25.548524    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" path="/var/lib/kubelet/pods/d7d700c8-9e21-4d93-964f-aad473eb5dcd/volumes"
Nov 02 18:07:26.298609 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:26.298574    5042 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-a8e574/agc-cred-c7aa0fd5ea16: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:26.298609 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:26.298606    5042 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7: secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:26.298951 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:26.298671    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials podName:277e3f59-79bb-4760-9513-b8c1583ba28a nodeName:}" failed. No retries permitted until 2026-11-02 18:07:42.298653098 +0000 UTC m=+2074.878062483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a") : secret "agc-cred-c7aa0fd5ea16" not found
Nov 02 18:07:27.456863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.455914    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" podUID="277e3f59-79bb-4760-9513-b8c1583ba28a" containerName="payload-processing" containerID="cri-o://07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9" gracePeriod=25
Nov 02 18:07:27.783008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.782974    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:27.857311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.857279    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k\") pod \"277e3f59-79bb-4760-9513-b8c1583ba28a\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") "
Nov 02 18:07:27.857546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.857368    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume\") pod \"277e3f59-79bb-4760-9513-b8c1583ba28a\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") "
Nov 02 18:07:27.857546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.857536    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume\") pod \"277e3f59-79bb-4760-9513-b8c1583ba28a\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") "
Nov 02 18:07:27.857702 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.857602    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials\") pod \"277e3f59-79bb-4760-9513-b8c1583ba28a\" (UID: \"277e3f59-79bb-4760-9513-b8c1583ba28a\") "
Nov 02 18:07:27.858207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.858040    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume" pod "277e3f59-79bb-4760-9513-b8c1583ba28a" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:27.858207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.858156    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume" pod "277e3f59-79bb-4760-9513-b8c1583ba28a" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:27.858667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.858642    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:27.858758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.858671    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277e3f59-79bb-4760-9513-b8c1583ba28a-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:27.860085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.860059    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k" pod "277e3f59-79bb-4760-9513-b8c1583ba28a" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a"). InnerVolumeSpecName "kube-api-access-x277k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:27.860886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.860734    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials" pod "277e3f59-79bb-4760-9513-b8c1583ba28a" (UID: "277e3f59-79bb-4760-9513-b8c1583ba28a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:27.960797 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.960762    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:27.960797 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.960797    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x277k\" (UniqueName: \"kubernetes.io/projected/277e3f59-79bb-4760-9513-b8c1583ba28a-kube-api-access-x277k\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:27.992381 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.992349    5042 generic.go:356] "Generic (PLEG): container finished" podID="277e3f59-79bb-4760-9513-b8c1583ba28a" containerID="07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9" exitCode=0
Nov 02 18:07:27.992586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.992396    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"
Nov 02 18:07:27.992586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.992431    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" event={"ID":"277e3f59-79bb-4760-9513-b8c1583ba28a","Type":"ContainerDied","Data":"07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9"}
Nov 02 18:07:27.992586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.992490    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7" event={"ID":"277e3f59-79bb-4760-9513-b8c1583ba28a","Type":"ContainerDied","Data":"064961845d7a4dd319f5c15d894a35d23d2cbe8629c7274963fad21ed7a7facb"}
Nov 02 18:07:27.992586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:27.992514    5042 scope.go:122] "RemoveContainer" containerID="07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9"
Nov 02 18:07:28.008230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:28.008204    5042 scope.go:122] "RemoveContainer" containerID="07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9"
Nov 02 18:07:28.008612 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:28.008586    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9\": container with ID starting with 07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9 not found: ID does not exist" containerID="07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9"
Nov 02 18:07:28.008748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:28.008621    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9"} err="failed to get container status \"07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9\": rpc error: code = NotFound desc = could not find container \"07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9\": container with ID starting with 07402ae036ea7709c24611c22f578189212767a6f6b7d10a5bb34da3042207a9 not found: ID does not exist"
Nov 02 18:07:28.019501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:28.019470    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"]
Nov 02 18:07:28.024992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:28.024964    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a8e574/payload-processing-external-model-e2e-extmod-a-a8e574-6dbd2nrm7"]
Nov 02 18:07:29.546434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.546394    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277e3f59-79bb-4760-9513-b8c1583ba28a" path="/var/lib/kubelet/pods/277e3f59-79bb-4760-9513-b8c1583ba28a/volumes"
Nov 02 18:07:29.875055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.875025    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.910686    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.910745    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.910831    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.910894    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.910949    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.911030    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.911073    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.911136    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.911224    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token\") pod \"7fb7660e-88b1-4348-9076-2a863a400210\" (UID: \"7fb7660e-88b1-4348-9076-2a863a400210\") "
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.913208    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.913404    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.913804    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.914006    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.923216    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.923395    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.923461    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "kube-api-access-8msvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:07:29.923583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.923534    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:07:29.925226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:29.925206    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy" pod "7fb7660e-88b1-4348-9076-2a863a400210" (UID: "7fb7660e-88b1-4348-9076-2a863a400210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:07:30.032858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032807    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8msvh\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-kube-api-access-8msvh\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.032858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032859    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb7660e-88b1-4348-9076-2a863a400210-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032874    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb7660e-88b1-4348-9076-2a863a400210-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032887    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032899    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032907    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032915    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb7660e-88b1-4348-9076-2a863a400210-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032923    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.033103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.032932    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb7660e-88b1-4348-9076-2a863a400210-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:07:30.037686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.037659    5042 generic.go:356] "Generic (PLEG): container finished" podID="7fb7660e-88b1-4348-9076-2a863a400210" containerID="7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b" exitCode=0
Nov 02 18:07:30.037866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.037773    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"
Nov 02 18:07:30.037866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.037797    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" event={"ID":"7fb7660e-88b1-4348-9076-2a863a400210","Type":"ContainerDied","Data":"7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b"}
Nov 02 18:07:30.038001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.037869    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq" event={"ID":"7fb7660e-88b1-4348-9076-2a863a400210","Type":"ContainerDied","Data":"6e6a66892c04ce63a40785115cd3726e3c87a71b98b293a18bc892cbb6b49cfd"}
Nov 02 18:07:30.038001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.037894    5042 scope.go:122] "RemoveContainer" containerID="7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b"
Nov 02 18:07:30.050635 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.050615    5042 scope.go:122] "RemoveContainer" containerID="7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b"
Nov 02 18:07:30.050967 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:30.050947    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b\": container with ID starting with 7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b not found: ID does not exist" containerID="7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b"
Nov 02 18:07:30.051026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.050976    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b"} err="failed to get container status \"7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b\": rpc error: code = NotFound desc = could not find container \"7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b\": container with ID starting with 7ff38707c0ec87fbbfe0711346195b86cb7b649b1bb7ec5e7f640c717527793b not found: ID does not exist"
Nov 02 18:07:30.065242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.065205    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"]
Nov 02 18:07:30.070248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:30.070221    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-2wrzq"]
Nov 02 18:07:31.547156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:31.547125    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb7660e-88b1-4348-9076-2a863a400210" path="/var/lib/kubelet/pods/7fb7660e-88b1-4348-9076-2a863a400210/volumes"
Nov 02 18:07:40.821038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.817239    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"]
Nov 02 18:07:40.821038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.818563    5042 cpu_manager.go:414] "containerMap: removing container" podUID="277e3f59-79bb-4760-9513-b8c1583ba28a" containerName="payload-processing"
Nov 02 18:07:40.821038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.818586    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277e3f59-79bb-4760-9513-b8c1583ba28a" containerName="payload-processing"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825106    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825127    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825150    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825157    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825197    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" containerName="revoke-keys"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825204    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" containerName="revoke-keys"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825456    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="277e3f59-79bb-4760-9513-b8c1583ba28a" containerName="payload-processing"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825471    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d700c8-9e21-4d93-964f-aad473eb5dcd" containerName="istio-proxy"
Nov 02 18:07:40.825476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825484    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb7660e-88b1-4348-9076-2a863a400210" containerName="istio-proxy"
Nov 02 18:07:40.826455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.825499    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" containerName="revoke-keys"
Nov 02 18:07:40.839954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.831664    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"]
Nov 02 18:07:40.839954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.831815    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:40.839954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.839261    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2db162-serving-cert\""
Nov 02 18:07:40.898616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.898562    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:40.898828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.898733    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvb7p\" (UniqueName: \"kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:40.898828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.898787    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:40.899121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:40.899083    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.000966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.000924    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.001127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.001085    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.001244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.001232    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.001319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.001303    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvb7p\" (UniqueName: \"kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.001885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.001861    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.003730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.003710    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.003938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.003920    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.009968 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.009939    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvb7p\" (UniqueName: \"kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p\") pod \"maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.166241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.166147    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:41.596458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:41.596429    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"]
Nov 02 18:07:42.211199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:42.211162    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw" event={"ID":"2254cf9a-58ee-46db-86dd-e17faaee6caf","Type":"ContainerStarted","Data":"b2069988705bb10f4dc97594a14fb316a0737a820ed41214a4438098b04fed9b"}
Nov 02 18:07:43.223951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:43.223910    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw" event={"ID":"2254cf9a-58ee-46db-86dd-e17faaee6caf","Type":"ContainerStarted","Data":"7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531"}
Nov 02 18:07:43.224418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:43.224033    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:43.248251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:43.248186    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw" podStartSLOduration=2.695181279 podStartE2EDuration="3.248167014s" totalImagesPullingTime="552.985735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:43.242447543 +0000 UTC m=+2075.821856931" watchObservedRunningTime="2026-11-02 18:07:43.248167014 +0000 UTC m=+2075.827576402"
Nov 02 18:07:48.549656 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:07:48.549620    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:07:49.242630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:49.242599    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:07:57.569019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:57.568986    5042 generic.go:356] "Generic (PLEG): container finished" podID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerID="c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312" exitCode=0
Nov 02 18:07:57.569516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:57.569055    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" event={"ID":"dd705464-df0f-4d2f-bb88-3f9ef9057ad6","Type":"ContainerDied","Data":"c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"}
Nov 02 18:07:57.569595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:57.569580    5042 scope.go:122] "RemoveContainer" containerID="c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"
Nov 02 18:07:57.571618 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:57.571584    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:57.571747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:57.571676    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:57.571819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:57.571747    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:58.086902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.086798    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"]
Nov 02 18:07:58.107899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.107813    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.118972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.117771    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"]
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131521    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131620    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131652    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131677    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v955\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131714    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131804    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131826    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131919    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.134113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.131946    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.232905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.232818    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.232994    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233047    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233071    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v955\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233125    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233255    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233287    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233343    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233377    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233513    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233560    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233593    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.233892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.233871    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.234572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.234550    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.235990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.235964    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.236180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.236154    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.242257 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.242233    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.242440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.242422    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v955\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955\") pod \"e2e-extmod-a-42d786-openshift-default-776544c777-97z24\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") " pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.440155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.440118    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:07:58.623751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.623706    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" event={"ID":"dd705464-df0f-4d2f-bb88-3f9ef9057ad6","Type":"ContainerStarted","Data":"a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4"}
Nov 02 18:07:58.879805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.878363    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:07:58.887991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.887954    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:07:58.902088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:58.902040    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:07:59.113862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.113646    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"]
Nov 02 18:07:59.214236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.214189    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:59.232166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.232122    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:59.233888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.232435    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:07:59.662963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.662927    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" event={"ID":"a8b919ab-89b4-492b-a4fa-532cfb80bc48","Type":"ContainerStarted","Data":"54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94"}
Nov 02 18:07:59.663571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.663547    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" event={"ID":"a8b919ab-89b4-492b-a4fa-532cfb80bc48","Type":"ContainerStarted","Data":"ad7c0f40408d2e9e0c0314576312efb9d2c926d169d823d331cb2e3101d42dfd"}
Nov 02 18:07:59.698864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.695555    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podStartSLOduration=1.695534189 podStartE2EDuration="1.695534189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:59.684484071 +0000 UTC m=+2092.263893458" watchObservedRunningTime="2026-11-02 18:07:59.695534189 +0000 UTC m=+2092.274943575"
Nov 02 18:07:59.870214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.870161    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:07:59.870423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:07:59.870233    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:00.443862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:00.443504    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:08:00.443862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:00.443681    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:00.443862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:00.443772    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:00.884925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:00.882252    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:08:00.884925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:00.882313    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:00.890928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:00.885914    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:08:01.371078 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.365070    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:01.463867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.457974    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:01.463867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.458049    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:01.499875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.480481    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:01.499875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.480832    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 18:08:01.553866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.553127    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:01.889867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.885302    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:08:01.889867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:01.885373    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:02.027863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.027591    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:02.082865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.081935    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:02.164924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.161879    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:08:02.164924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.162299    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:02.225179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.224895    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:02.281054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.281008    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:02.442958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.442921    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:02.443241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.443210    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:02.887916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.878349    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:08:02.887916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:02.878411    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:03.449534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:03.449499    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:03.450173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:03.450139    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:03.870969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:03.870878    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:08:03.870969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:03.870932    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:04.441475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:04.441434    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:04.441701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:04.441491    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:04.870277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:04.870185    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:08:04.870866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:04.870796    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:05.441275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:05.441228    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:05.441275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:05.441278    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:05.870539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:05.870452    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Nov 02 18:08:05.870539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:05.870510    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Nov 02 18:08:06.443382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:06.443343    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:06.443707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:06.443678    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:07.236217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:07.236171    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:08:07.238637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:07.238599    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:08:07.475873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:07.474778    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-42d786-openshift-default-776544c777-97z24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body=
Nov 02 18:08:07.475873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:07.474860    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused"
Nov 02 18:08:08.475050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:08.473208    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:08:08.475534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:08.475225    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:08:08.476341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:08.476317    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:08:09.458433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:09.458403    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:08:09.461517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:09.461495    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:08:09.464504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:09.464483    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:08:09.467502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:09.467487    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:08:18.544452 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:08:18.544366    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:08:32.422868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:32.422808    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"]
Nov 02 18:08:32.423954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:32.423914    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy" containerID="cri-o://5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530" gracePeriod=30
Nov 02 18:08:37.843506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.843473    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 18:08:37.977086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977052    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.977288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977121    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.977288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977155    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.977288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977196    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.977577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977314    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.977577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977511    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.977955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977833    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:08:37.977955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.977943    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.978125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.978035    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.978125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.978064    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs\") pod \"61a5e54c-3842-4947-859a-a4d1780467c6\" (UID: \"61a5e54c-3842-4947-859a-a4d1780467c6\") "
Nov 02 18:08:37.978620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.978580    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:08:37.979173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.978973    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:08:37.979282 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979203    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:08:37.979445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979403    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:08:37.979871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979741    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:37.979871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979769    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a5e54c-3842-4947-859a-a4d1780467c6-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:37.979871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979783    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:37.979871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979799    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:37.979871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.979814    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:37.981315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.981293    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:08:37.982125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.982097    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:08:37.983097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.983066    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "kube-api-access-zfnwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:08:37.983373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:37.983307    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token" pod "61a5e54c-3842-4947-859a-a4d1780467c6" (UID: "61a5e54c-3842-4947-859a-a4d1780467c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:08:38.081953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.081853    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a5e54c-3842-4947-859a-a4d1780467c6-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:38.081953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.081894    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a5e54c-3842-4947-859a-a4d1780467c6-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:38.081953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.081907    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnwt\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-kube-api-access-zfnwt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:38.081953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.081917    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a5e54c-3842-4947-859a-a4d1780467c6-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:08:38.531101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.531063    5042 generic.go:356] "Generic (PLEG): container finished" podID="61a5e54c-3842-4947-859a-a4d1780467c6" containerID="5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530" exitCode=0
Nov 02 18:08:38.531301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.531114    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"
Nov 02 18:08:38.531301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.531148    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" event={"ID":"61a5e54c-3842-4947-859a-a4d1780467c6","Type":"ContainerDied","Data":"5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530"}
Nov 02 18:08:38.531301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.531209    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm" event={"ID":"61a5e54c-3842-4947-859a-a4d1780467c6","Type":"ContainerDied","Data":"c56f959152537f76cfddc471aaabcc5b51ff5fadb26740d001d870496a25e738"}
Nov 02 18:08:38.531301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.531233    5042 scope.go:122] "RemoveContainer" containerID="5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530"
Nov 02 18:08:38.549935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.549757    5042 scope.go:122] "RemoveContainer" containerID="999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"
Nov 02 18:08:38.561783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.561742    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"]
Nov 02 18:08:38.562679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.562661    5042 scope.go:122] "RemoveContainer" containerID="5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530"
Nov 02 18:08:38.563075 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:08:38.563048    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530\": container with ID starting with 5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530 not found: ID does not exist" containerID="5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530"
Nov 02 18:08:38.563175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.563082    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530"} err="failed to get container status \"5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530\": rpc error: code = NotFound desc = could not find container \"5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530\": container with ID starting with 5158a06408130a1b728f2876de9243fedf0449a688f2727432028680f7070530 not found: ID does not exist"
Nov 02 18:08:38.563175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.563103    5042 scope.go:122] "RemoveContainer" containerID="999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"
Nov 02 18:08:38.563412 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:08:38.563394    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e\": container with ID starting with 999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e not found: ID does not exist" containerID="999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"
Nov 02 18:08:38.563478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.563418    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e"} err="failed to get container status \"999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e\": rpc error: code = NotFound desc = could not find container \"999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e\": container with ID starting with 999ee046b3797106e4b98cd8f1e87a4a75785ec5bb58198ed821835cfaa8dc9e not found: ID does not exist"
Nov 02 18:08:38.567107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:38.567084    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-5pshm"]
Nov 02 18:08:39.546763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:08:39.546730    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" path="/var/lib/kubelet/pods/61a5e54c-3842-4947-859a-a4d1780467c6/volumes"
Nov 02 18:08:48.540819 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:08:48.540785    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:09:02.599703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.599666    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"]
Nov 02 18:09:02.600566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.600545    5042 cpu_manager.go:414] "containerMap: removing container" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy"
Nov 02 18:09:02.600566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.600564    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy"
Nov 02 18:09:02.600740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.600634    5042 cpu_manager.go:414] "containerMap: removing container" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy"
Nov 02 18:09:02.600740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.600640    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy"
Nov 02 18:09:02.600811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.600770    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy"
Nov 02 18:09:02.600811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.600786    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a5e54c-3842-4947-859a-a4d1780467c6" containerName="istio-proxy"
Nov 02 18:09:02.604632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.604615    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.615018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.614992    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:09:02.615329 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.615311    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:09:02.624455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.624422    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"]
Nov 02 18:09:02.801713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.801680    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.801914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.801730    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.801984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.801941    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfcb\" (UniqueName: \"kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.903953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.903855    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.903953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.903910    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.904226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.904016    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfcb\" (UniqueName: \"kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.904719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.904693    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.906794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.906771    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.917482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.917453    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfcb\" (UniqueName: \"kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb\") pod \"maas-api-revoke-sub-25386faa-rfm9f\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:02.919616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:02.919593    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:03.129554 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.129521    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"]
Nov 02 18:09:03.134499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.134475    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.145760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.145725    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"]
Nov 02 18:09:03.212307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.212275    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"]
Nov 02 18:09:03.313386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.313345    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.313611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.313428    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sssgx\" (UniqueName: \"kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.313663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.313630    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.416100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.416057    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.416305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.416160    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sssgx\" (UniqueName: \"kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.416381 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.416342    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.418156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.417995    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.473303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.473211    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.473503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.473309    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sssgx\" (UniqueName: \"kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx\") pod \"maas-api-revoke-sub-ee20b4b6-xrfp7\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.762934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.762787    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:03.932714 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.932668    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f" event={"ID":"283efc9e-0afe-4e39-8674-e2943a39e96d","Type":"ContainerStarted","Data":"db368686f993616661e4ee1173fdc095a19c43f8b6d4f70feaf7c81d57c067f4"}
Nov 02 18:09:03.932953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.932739    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f" event={"ID":"283efc9e-0afe-4e39-8674-e2943a39e96d","Type":"ContainerStarted","Data":"5607445a001cf0b1c888d4fcdc392d88157748a7063d223f5063a45dd827c8d1"}
Nov 02 18:09:03.952559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:03.952500    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f" podStartSLOduration=1.952485039 podStartE2EDuration="1.952485039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:03.950003967 +0000 UTC m=+2156.529413354" watchObservedRunningTime="2026-11-02 18:09:03.952485039 +0000 UTC m=+2156.531894424"
Nov 02 18:09:04.063115 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:09:04.063079    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911da677_275c_4c93_b399_90a2a4098f54.slice/crio-4b8a11f5d32c97f467180c2e17bc3f48974fdd48df7e34dc1dde504588aa8ceb WatchSource:0}: Error finding container 4b8a11f5d32c97f467180c2e17bc3f48974fdd48df7e34dc1dde504588aa8ceb: Status 404 returned error can't find the container with id 4b8a11f5d32c97f467180c2e17bc3f48974fdd48df7e34dc1dde504588aa8ceb
Nov 02 18:09:04.063655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:04.063585    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"]
Nov 02 18:09:04.944615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:04.944556    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7" event={"ID":"911da677-275c-4c93-b399-90a2a4098f54","Type":"ContainerStarted","Data":"baba4c48286127e54e3b8f5f42f59a4418264b3461b103d1f8b7e92711ebf712"}
Nov 02 18:09:04.945165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:04.944631    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7" event={"ID":"911da677-275c-4c93-b399-90a2a4098f54","Type":"ContainerStarted","Data":"4b8a11f5d32c97f467180c2e17bc3f48974fdd48df7e34dc1dde504588aa8ceb"}
Nov 02 18:09:04.947077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:04.947048    5042 generic.go:356] "Generic (PLEG): container finished" podID="283efc9e-0afe-4e39-8674-e2943a39e96d" containerID="db368686f993616661e4ee1173fdc095a19c43f8b6d4f70feaf7c81d57c067f4" exitCode=0
Nov 02 18:09:04.947219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:04.947127    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f" event={"ID":"283efc9e-0afe-4e39-8674-e2943a39e96d","Type":"ContainerDied","Data":"db368686f993616661e4ee1173fdc095a19c43f8b6d4f70feaf7c81d57c067f4"}
Nov 02 18:09:04.969369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:04.969315    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7" podStartSLOduration=1.969301682 podStartE2EDuration="1.969301682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:04.964935631 +0000 UTC m=+2157.544345019" watchObservedRunningTime="2026-11-02 18:09:04.969301682 +0000 UTC m=+2157.548711101"
Nov 02 18:09:05.957672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:05.957636    5042 generic.go:356] "Generic (PLEG): container finished" podID="911da677-275c-4c93-b399-90a2a4098f54" containerID="baba4c48286127e54e3b8f5f42f59a4418264b3461b103d1f8b7e92711ebf712" exitCode=0
Nov 02 18:09:05.958129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:05.957722    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7" event={"ID":"911da677-275c-4c93-b399-90a2a4098f54","Type":"ContainerDied","Data":"baba4c48286127e54e3b8f5f42f59a4418264b3461b103d1f8b7e92711ebf712"}
Nov 02 18:09:06.166374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.166344    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:06.277438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.277343    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb\" (UniqueName: \"kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb\") pod \"283efc9e-0afe-4e39-8674-e2943a39e96d\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") "
Nov 02 18:09:06.277625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.277607    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca\") pod \"283efc9e-0afe-4e39-8674-e2943a39e96d\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") "
Nov 02 18:09:06.277695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.277680    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca\") pod \"283efc9e-0afe-4e39-8674-e2943a39e96d\" (UID: \"283efc9e-0afe-4e39-8674-e2943a39e96d\") "
Nov 02 18:09:06.278040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.278009    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca" pod "283efc9e-0afe-4e39-8674-e2943a39e96d" (UID: "283efc9e-0afe-4e39-8674-e2943a39e96d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:09:06.278569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.278550    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:06.279825 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.279800    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb" pod "283efc9e-0afe-4e39-8674-e2943a39e96d" (UID: "283efc9e-0afe-4e39-8674-e2943a39e96d"). InnerVolumeSpecName "kube-api-access-msfcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:09:06.280477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.280451    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca" pod "283efc9e-0afe-4e39-8674-e2943a39e96d" (UID: "283efc9e-0afe-4e39-8674-e2943a39e96d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:06.380428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.380385    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283efc9e-0afe-4e39-8674-e2943a39e96d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:06.380428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.380415    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfcb\" (UniqueName: \"kubernetes.io/projected/283efc9e-0afe-4e39-8674-e2943a39e96d-kube-api-access-msfcb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:06.968576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.968542    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"
Nov 02 18:09:06.969143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.968573    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f" event={"ID":"283efc9e-0afe-4e39-8674-e2943a39e96d","Type":"ContainerDied","Data":"5607445a001cf0b1c888d4fcdc392d88157748a7063d223f5063a45dd827c8d1"}
Nov 02 18:09:06.969143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:06.968622    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5607445a001cf0b1c888d4fcdc392d88157748a7063d223f5063a45dd827c8d1"
Nov 02 18:09:07.179056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.179030    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:07.220928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.217597    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca\") pod \"911da677-275c-4c93-b399-90a2a4098f54\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") "
Nov 02 18:09:07.220928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.217677    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca\") pod \"911da677-275c-4c93-b399-90a2a4098f54\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") "
Nov 02 18:09:07.220928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.217994    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx\" (UniqueName: \"kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx\") pod \"911da677-275c-4c93-b399-90a2a4098f54\" (UID: \"911da677-275c-4c93-b399-90a2a4098f54\") "
Nov 02 18:09:07.227872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.223298    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca" pod "911da677-275c-4c93-b399-90a2a4098f54" (UID: "911da677-275c-4c93-b399-90a2a4098f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:09:07.257956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.247332    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx" pod "911da677-275c-4c93-b399-90a2a4098f54" (UID: "911da677-275c-4c93-b399-90a2a4098f54"). InnerVolumeSpecName "kube-api-access-sssgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:09:07.257956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.251308    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca" pod "911da677-275c-4c93-b399-90a2a4098f54" (UID: "911da677-275c-4c93-b399-90a2a4098f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:07.321221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.320046    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911da677-275c-4c93-b399-90a2a4098f54-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:07.321221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.320077    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911da677-275c-4c93-b399-90a2a4098f54-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:07.321221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.320088    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sssgx\" (UniqueName: \"kubernetes.io/projected/911da677-275c-4c93-b399-90a2a4098f54-kube-api-access-sssgx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:07.980113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.980078    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"
Nov 02 18:09:07.980113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.980088    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7" event={"ID":"911da677-275c-4c93-b399-90a2a4098f54","Type":"ContainerDied","Data":"4b8a11f5d32c97f467180c2e17bc3f48974fdd48df7e34dc1dde504588aa8ceb"}
Nov 02 18:09:07.980545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:07.980140    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8a11f5d32c97f467180c2e17bc3f48974fdd48df7e34dc1dde504588aa8ceb"
Nov 02 18:09:13.995008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.994968    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"]
Nov 02 18:09:13.996269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.996244    5042 cpu_manager.go:414] "containerMap: removing container" podUID="911da677-275c-4c93-b399-90a2a4098f54" containerName="revoke-keys"
Nov 02 18:09:13.996385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.996274    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911da677-275c-4c93-b399-90a2a4098f54" containerName="revoke-keys"
Nov 02 18:09:13.996385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.996372    5042 cpu_manager.go:414] "containerMap: removing container" podUID="283efc9e-0afe-4e39-8674-e2943a39e96d" containerName="revoke-keys"
Nov 02 18:09:13.996385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.996382    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283efc9e-0afe-4e39-8674-e2943a39e96d" containerName="revoke-keys"
Nov 02 18:09:13.996607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.996592    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="283efc9e-0afe-4e39-8674-e2943a39e96d" containerName="revoke-keys"
Nov 02 18:09:13.996666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:13.996613    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="911da677-275c-4c93-b399-90a2a4098f54" containerName="revoke-keys"
Nov 02 18:09:14.007361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.007322    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.017817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.017778    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"]
Nov 02 18:09:14.018023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.018004    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:09:14.018123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.018107    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:09:14.053003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.052970    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.053178 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.053122    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdgt\" (UniqueName: \"kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.053288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.053275    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.222864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.215106    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.222864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.215265    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdgt\" (UniqueName: \"kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.222864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.215449    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.258938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.246677    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.295874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.288307    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.431908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.412249    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdgt\" (UniqueName: \"kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt\") pod \"maas-api-revoke-sub-0e34f552-bdslm\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:14.650237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:14.650149    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:15.122893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.122864    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"]
Nov 02 18:09:15.133995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.133967    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"]
Nov 02 18:09:15.134279 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.134098    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.189450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.189393    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.189655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.189562    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmskg\" (UniqueName: \"kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.192245 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.192206    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.293949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.293900    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.294149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.294040    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmskg\" (UniqueName: \"kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.294231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.294212    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.294985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.294959    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.297163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.297133    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.305201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.305171    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmskg\" (UniqueName: \"kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg\") pod \"maas-api-revoke-sub-d53691d9-bf9m4\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.422918 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:09:15.422874    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd445901d_dd96_4bdf_9bc1_c95bb7c7b301.slice/crio-10c2c4cd2a526d29acf75943b90f4df2b77c10021ab530a5dc2aaf404d7b92a9 WatchSource:0}: Error finding container 10c2c4cd2a526d29acf75943b90f4df2b77c10021ab530a5dc2aaf404d7b92a9: Status 404 returned error can't find the container with id 10c2c4cd2a526d29acf75943b90f4df2b77c10021ab530a5dc2aaf404d7b92a9
Nov 02 18:09:15.424435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.424404    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"]
Nov 02 18:09:15.488471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.488405    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:15.800988 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:09:15.800947    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd21099_9f25_426d_8b01_22d2547ec98a.slice/crio-65fe509d591dd81899923162fde09f7666f4645cd5b8a15331635f9e46e52c97 WatchSource:0}: Error finding container 65fe509d591dd81899923162fde09f7666f4645cd5b8a15331635f9e46e52c97: Status 404 returned error can't find the container with id 65fe509d591dd81899923162fde09f7666f4645cd5b8a15331635f9e46e52c97
Nov 02 18:09:15.802684 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:15.802658    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"]
Nov 02 18:09:16.151321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:16.151278    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm" event={"ID":"d445901d-dd96-4bdf-9bc1-c95bb7c7b301","Type":"ContainerStarted","Data":"b160b69a14c8cf90daa6ed834e9b8a9e29a18291973756230eefdaa3389ec84e"}
Nov 02 18:09:16.151829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:16.151349    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm" event={"ID":"d445901d-dd96-4bdf-9bc1-c95bb7c7b301","Type":"ContainerStarted","Data":"10c2c4cd2a526d29acf75943b90f4df2b77c10021ab530a5dc2aaf404d7b92a9"}
Nov 02 18:09:16.153157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:16.153128    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4" event={"ID":"cdd21099-9f25-426d-8b01-22d2547ec98a","Type":"ContainerStarted","Data":"65fe509d591dd81899923162fde09f7666f4645cd5b8a15331635f9e46e52c97"}
Nov 02 18:09:16.182898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:16.182823    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm" podStartSLOduration=3.182807441 podStartE2EDuration="3.182807441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:16.181054229 +0000 UTC m=+2168.760463617" watchObservedRunningTime="2026-11-02 18:09:16.182807441 +0000 UTC m=+2168.762216826"
Nov 02 18:09:17.166464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:17.166426    5042 generic.go:356] "Generic (PLEG): container finished" podID="d445901d-dd96-4bdf-9bc1-c95bb7c7b301" containerID="b160b69a14c8cf90daa6ed834e9b8a9e29a18291973756230eefdaa3389ec84e" exitCode=0
Nov 02 18:09:17.166464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:17.166470    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm" event={"ID":"d445901d-dd96-4bdf-9bc1-c95bb7c7b301","Type":"ContainerDied","Data":"b160b69a14c8cf90daa6ed834e9b8a9e29a18291973756230eefdaa3389ec84e"}
Nov 02 18:09:17.168674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:17.168646    5042 generic.go:356] "Generic (PLEG): container finished" podID="cdd21099-9f25-426d-8b01-22d2547ec98a" containerID="e69714141d5aea66433b475d7bd1c3da5d22268871bd7e051a01b1eed2c99296" exitCode=0
Nov 02 18:09:17.168812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:17.168721    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4" event={"ID":"cdd21099-9f25-426d-8b01-22d2547ec98a","Type":"ContainerDied","Data":"e69714141d5aea66433b475d7bd1c3da5d22268871bd7e051a01b1eed2c99296"}
Nov 02 18:09:18.436884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.436795    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:18.443440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.443414    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:18.482126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.482089    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\") pod \"cdd21099-9f25-426d-8b01-22d2547ec98a\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") "
Nov 02 18:09:18.482377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.482361    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg\" (UniqueName: \"kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg\") pod \"cdd21099-9f25-426d-8b01-22d2547ec98a\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") "
Nov 02 18:09:18.482494 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.482480    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca\") pod \"cdd21099-9f25-426d-8b01-22d2547ec98a\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") "
Nov 02 18:09:18.482720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.482707    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt\" (UniqueName: \"kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt\") pod \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") "
Nov 02 18:09:18.483091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.483075    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca\") pod \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") "
Nov 02 18:09:18.483251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.483235    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca\") pod \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\" (UID: \"d445901d-dd96-4bdf-9bc1-c95bb7c7b301\") "
Nov 02 18:09:18.484337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.484310    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca" pod "cdd21099-9f25-426d-8b01-22d2547ec98a" (UID: "cdd21099-9f25-426d-8b01-22d2547ec98a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:09:18.486513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.486489    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca" pod "d445901d-dd96-4bdf-9bc1-c95bb7c7b301" (UID: "d445901d-dd96-4bdf-9bc1-c95bb7c7b301"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:09:18.593675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.585333    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca" pod "cdd21099-9f25-426d-8b01-22d2547ec98a" (UID: "cdd21099-9f25-426d-8b01-22d2547ec98a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:18.593675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.585682    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\") pod \"cdd21099-9f25-426d-8b01-22d2547ec98a\" (UID: \"cdd21099-9f25-426d-8b01-22d2547ec98a\") "
Nov 02 18:09:18.593675 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:09:18.586580    5042 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/cdd21099-9f25-426d-8b01-22d2547ec98a/volumes/kubernetes.io~secret/maas-api-xks-ca
Nov 02 18:09:18.593675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.586602    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca" pod "cdd21099-9f25-426d-8b01-22d2547ec98a" (UID: "cdd21099-9f25-426d-8b01-22d2547ec98a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:18.593675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.587614    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg" pod "cdd21099-9f25-426d-8b01-22d2547ec98a" (UID: "cdd21099-9f25-426d-8b01-22d2547ec98a"). InnerVolumeSpecName "kube-api-access-fmskg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:09:18.594747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.594246    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt" pod "d445901d-dd96-4bdf-9bc1-c95bb7c7b301" (UID: "d445901d-dd96-4bdf-9bc1-c95bb7c7b301"). InnerVolumeSpecName "kube-api-access-6tdgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:09:18.597892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.596938    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:18.597892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.596972    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:18.597892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.596986    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmskg\" (UniqueName: \"kubernetes.io/projected/cdd21099-9f25-426d-8b01-22d2547ec98a-kube-api-access-fmskg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:18.597892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.597001    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdd21099-9f25-426d-8b01-22d2547ec98a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:18.597892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.597014    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tdgt\" (UniqueName: \"kubernetes.io/projected/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-kube-api-access-6tdgt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:18.598951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.598832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca" pod "d445901d-dd96-4bdf-9bc1-c95bb7c7b301" (UID: "d445901d-dd96-4bdf-9bc1-c95bb7c7b301"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:18.632686 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:09:18.632650    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:09:18.698606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:18.698573    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d445901d-dd96-4bdf-9bc1-c95bb7c7b301-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:19.189786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:19.189752    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"
Nov 02 18:09:19.189995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:19.189782    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm" event={"ID":"d445901d-dd96-4bdf-9bc1-c95bb7c7b301","Type":"ContainerDied","Data":"10c2c4cd2a526d29acf75943b90f4df2b77c10021ab530a5dc2aaf404d7b92a9"}
Nov 02 18:09:19.189995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:19.189886    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10c2c4cd2a526d29acf75943b90f4df2b77c10021ab530a5dc2aaf404d7b92a9"
Nov 02 18:09:19.191980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:19.191935    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"
Nov 02 18:09:19.192131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:19.191982    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4" event={"ID":"cdd21099-9f25-426d-8b01-22d2547ec98a","Type":"ContainerDied","Data":"65fe509d591dd81899923162fde09f7666f4645cd5b8a15331635f9e46e52c97"}
Nov 02 18:09:19.192131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:19.192032    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65fe509d591dd81899923162fde09f7666f4645cd5b8a15331635f9e46e52c97"
Nov 02 18:09:35.114866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:35.113954    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:09:37.105493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.105456    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"]
Nov 02 18:09:37.106466 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.106446    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cdd21099-9f25-426d-8b01-22d2547ec98a" containerName="revoke-keys"
Nov 02 18:09:37.106535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.106468    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd21099-9f25-426d-8b01-22d2547ec98a" containerName="revoke-keys"
Nov 02 18:09:37.106535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.106483    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d445901d-dd96-4bdf-9bc1-c95bb7c7b301" containerName="revoke-keys"
Nov 02 18:09:37.106535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.106490    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d445901d-dd96-4bdf-9bc1-c95bb7c7b301" containerName="revoke-keys"
Nov 02 18:09:37.106697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.106617    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd21099-9f25-426d-8b01-22d2547ec98a" containerName="revoke-keys"
Nov 02 18:09:37.106697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.106627    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d445901d-dd96-4bdf-9bc1-c95bb7c7b301" containerName="revoke-keys"
Nov 02 18:09:37.110949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.110927    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.115401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.115373    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:09:37.115578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.115427    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:09:37.121609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.121580    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"]
Nov 02 18:09:37.187926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.187367    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.187926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.187488    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.187926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.187670    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkq78\" (UniqueName: \"kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.289271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.289232    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.289479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.289321    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.289546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.289483    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkq78\" (UniqueName: \"kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.289938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.289916    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.292282 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.292258    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.298421 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.298401    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkq78\" (UniqueName: \"kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78\") pod \"maas-api-revoke-sub-5d46fb37-j8s7m\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.433479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.433377    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:37.731122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:37.731083    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"]
Nov 02 18:09:37.731534 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:09:37.731506    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6355fee8_e959_48dd_9b99_c1a9d9617ddd.slice/crio-e1a51ccc3c48817222052e84ca92ba6dfe45f4f06414934f436f428936d04780 WatchSource:0}: Error finding container e1a51ccc3c48817222052e84ca92ba6dfe45f4f06414934f436f428936d04780: Status 404 returned error can't find the container with id e1a51ccc3c48817222052e84ca92ba6dfe45f4f06414934f436f428936d04780
Nov 02 18:09:38.649900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:38.649791    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m" event={"ID":"6355fee8-e959-48dd-9b99-c1a9d9617ddd","Type":"ContainerStarted","Data":"bf1913b1e945acd4ec5329d8aa4bc2e275d2169e39f5eada2c3f363c3383454d"}
Nov 02 18:09:38.649900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:38.649888    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m" event={"ID":"6355fee8-e959-48dd-9b99-c1a9d9617ddd","Type":"ContainerStarted","Data":"e1a51ccc3c48817222052e84ca92ba6dfe45f4f06414934f436f428936d04780"}
Nov 02 18:09:38.673002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:38.672936    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m" podStartSLOduration=1.6729173880000001 podStartE2EDuration="1.672917388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:38.666315573 +0000 UTC m=+2191.245724959" watchObservedRunningTime="2026-11-02 18:09:38.672917388 +0000 UTC m=+2191.252326777"
Nov 02 18:09:39.662433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:39.662397    5042 generic.go:356] "Generic (PLEG): container finished" podID="6355fee8-e959-48dd-9b99-c1a9d9617ddd" containerID="bf1913b1e945acd4ec5329d8aa4bc2e275d2169e39f5eada2c3f363c3383454d" exitCode=0
Nov 02 18:09:39.662949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:39.662683    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m" event={"ID":"6355fee8-e959-48dd-9b99-c1a9d9617ddd","Type":"ContainerDied","Data":"bf1913b1e945acd4ec5329d8aa4bc2e275d2169e39f5eada2c3f363c3383454d"}
Nov 02 18:09:41.014551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.014479    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:41.092001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.091966    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca\") pod \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") "
Nov 02 18:09:41.092200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.092076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78\" (UniqueName: \"kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78\") pod \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") "
Nov 02 18:09:41.092245 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.092230    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca\") pod \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\" (UID: \"6355fee8-e959-48dd-9b99-c1a9d9617ddd\") "
Nov 02 18:09:41.093054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.093019    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca" pod "6355fee8-e959-48dd-9b99-c1a9d9617ddd" (UID: "6355fee8-e959-48dd-9b99-c1a9d9617ddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:09:41.094589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.094563    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca" pod "6355fee8-e959-48dd-9b99-c1a9d9617ddd" (UID: "6355fee8-e959-48dd-9b99-c1a9d9617ddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:41.094777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.094760    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78" pod "6355fee8-e959-48dd-9b99-c1a9d9617ddd" (UID: "6355fee8-e959-48dd-9b99-c1a9d9617ddd"). InnerVolumeSpecName "kube-api-access-mkq78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:09:41.196228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.196161    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:41.196510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.196485    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkq78\" (UniqueName: \"kubernetes.io/projected/6355fee8-e959-48dd-9b99-c1a9d9617ddd-kube-api-access-mkq78\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:41.196638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.196623    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6355fee8-e959-48dd-9b99-c1a9d9617ddd-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:41.736894 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.736828    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"
Nov 02 18:09:41.736894 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.736871    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m" event={"ID":"6355fee8-e959-48dd-9b99-c1a9d9617ddd","Type":"ContainerDied","Data":"e1a51ccc3c48817222052e84ca92ba6dfe45f4f06414934f436f428936d04780"}
Nov 02 18:09:41.737217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:41.736925    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a51ccc3c48817222052e84ca92ba6dfe45f4f06414934f436f428936d04780"
Nov 02 18:09:47.850810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.850735    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"]
Nov 02 18:09:47.851577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.851560    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6355fee8-e959-48dd-9b99-c1a9d9617ddd" containerName="revoke-keys"
Nov 02 18:09:47.851629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.851578    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6355fee8-e959-48dd-9b99-c1a9d9617ddd" containerName="revoke-keys"
Nov 02 18:09:47.851720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.851711    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6355fee8-e959-48dd-9b99-c1a9d9617ddd" containerName="revoke-keys"
Nov 02 18:09:47.856030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.856002    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:47.863225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.862911    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:09:47.863711 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.863689    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:09:47.864167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.864138    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"]
Nov 02 18:09:47.923202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.923153    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:47.923404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.923296    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:47.923591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:47.923565    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rkd\" (UniqueName: \"kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.024893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.024814    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.025088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.024926    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.025088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.025081    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rkd\" (UniqueName: \"kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.025685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.025659    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.027690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.027659    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.040019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.039990    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rkd\" (UniqueName: \"kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd\") pod \"maas-api-revoke-sub-adb0f52c-22nnm\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.178295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.178201    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:48.492598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.492566    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"]
Nov 02 18:09:48.493566 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:09:48.493534    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f35cdd5_7ada_4333_957e_a301964fbc49.slice/crio-af03c49640c1a12028a6847a8271459e46a48b61f122d69d992bfb3ae70287f3 WatchSource:0}: Error finding container af03c49640c1a12028a6847a8271459e46a48b61f122d69d992bfb3ae70287f3: Status 404 returned error can't find the container with id af03c49640c1a12028a6847a8271459e46a48b61f122d69d992bfb3ae70287f3
Nov 02 18:09:48.632677 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:09:48.632625    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:09:48.823195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:48.823148    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm" event={"ID":"5f35cdd5-7ada-4333-957e-a301964fbc49","Type":"ContainerStarted","Data":"af03c49640c1a12028a6847a8271459e46a48b61f122d69d992bfb3ae70287f3"}
Nov 02 18:09:49.836361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:49.836239    5042 generic.go:356] "Generic (PLEG): container finished" podID="5f35cdd5-7ada-4333-957e-a301964fbc49" containerID="30b39e8cb5800a6639b6c3e217c84df745d84e2a40be5c529dbffc240c1a942a" exitCode=0
Nov 02 18:09:49.836361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:49.836289    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm" event={"ID":"5f35cdd5-7ada-4333-957e-a301964fbc49","Type":"ContainerDied","Data":"30b39e8cb5800a6639b6c3e217c84df745d84e2a40be5c529dbffc240c1a942a"}
Nov 02 18:09:51.124514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.124433    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:51.228400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.228361    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca\") pod \"5f35cdd5-7ada-4333-957e-a301964fbc49\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") "
Nov 02 18:09:51.228588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.228562    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca\") pod \"5f35cdd5-7ada-4333-957e-a301964fbc49\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") "
Nov 02 18:09:51.228700 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.228665    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd\" (UniqueName: \"kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd\") pod \"5f35cdd5-7ada-4333-957e-a301964fbc49\" (UID: \"5f35cdd5-7ada-4333-957e-a301964fbc49\") "
Nov 02 18:09:51.230355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.230328    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca" pod "5f35cdd5-7ada-4333-957e-a301964fbc49" (UID: "5f35cdd5-7ada-4333-957e-a301964fbc49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:09:51.232098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.232073    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd" pod "5f35cdd5-7ada-4333-957e-a301964fbc49" (UID: "5f35cdd5-7ada-4333-957e-a301964fbc49"). InnerVolumeSpecName "kube-api-access-28rkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:09:51.233304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.233275    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca" pod "5f35cdd5-7ada-4333-957e-a301964fbc49" (UID: "5f35cdd5-7ada-4333-957e-a301964fbc49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:09:51.345870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.331541    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:51.346221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.346190    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f35cdd5-7ada-4333-957e-a301964fbc49-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:51.346343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.346331    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28rkd\" (UniqueName: \"kubernetes.io/projected/5f35cdd5-7ada-4333-957e-a301964fbc49-kube-api-access-28rkd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:09:51.858603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.858510    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm" event={"ID":"5f35cdd5-7ada-4333-957e-a301964fbc49","Type":"ContainerDied","Data":"af03c49640c1a12028a6847a8271459e46a48b61f122d69d992bfb3ae70287f3"}
Nov 02 18:09:51.858603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.858548    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"
Nov 02 18:09:51.858603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:51.858569    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af03c49640c1a12028a6847a8271459e46a48b61f122d69d992bfb3ae70287f3"
Nov 02 18:09:56.030268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:56.030229    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"]
Nov 02 18:09:56.035438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:56.035410    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b13d41-5cqxc"]
Nov 02 18:09:57.551419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:57.551372    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83aa7f6-4079-4822-b6b3-bd1b3ea69d09" path="/var/lib/kubelet/pods/f83aa7f6-4079-4822-b6b3-bd1b3ea69d09/volumes"
Nov 02 18:09:59.479866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:59.478609    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:09:59.795943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:09:59.789690    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Nov 02 18:10:01.388217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:01.388053    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"]
Nov 02 18:10:01.564748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:01.564713    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-058b8c50-8kjgk"]
Nov 02 18:10:03.091021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.090979    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"]
Nov 02 18:10:03.092226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.092200    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5f35cdd5-7ada-4333-957e-a301964fbc49" containerName="revoke-keys"
Nov 02 18:10:03.093068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.093047    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f35cdd5-7ada-4333-957e-a301964fbc49" containerName="revoke-keys"
Nov 02 18:10:03.095180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.095160    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f35cdd5-7ada-4333-957e-a301964fbc49" containerName="revoke-keys"
Nov 02 18:10:03.137562 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.136815    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.148247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.146522    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"]
Nov 02 18:10:03.148247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.147445    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:10:03.148934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.148831    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:10:03.243389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.243327    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kss\" (UniqueName: \"kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.243621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.243501    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.243621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.243536    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.345055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.344953    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kss\" (UniqueName: \"kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.345259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.345087    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.345259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.345117    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.346043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.346010    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.348059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.348029    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.354819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.354794    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kss\" (UniqueName: \"kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss\") pod \"maas-api-revoke-sub-13ff203f-nbc9t\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.468584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.468548    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:03.557710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.557317    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3977ce17-d1a7-4a34-a3a0-ddfb71a104ef" path="/var/lib/kubelet/pods/3977ce17-d1a7-4a34-a3a0-ddfb71a104ef/volumes"
Nov 02 18:10:03.762074 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:10:03.762038    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fe532f_9e72_4297_9278_d261e3e24111.slice/crio-e91d866993669d9bdc857abc36af61dc683a2fbe4571f8436b24155bed717d90 WatchSource:0}: Error finding container e91d866993669d9bdc857abc36af61dc683a2fbe4571f8436b24155bed717d90: Status 404 returned error can't find the container with id e91d866993669d9bdc857abc36af61dc683a2fbe4571f8436b24155bed717d90
Nov 02 18:10:03.763217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:03.763185    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"]
Nov 02 18:10:04.445559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:04.445512    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t" event={"ID":"39fe532f-9e72-4297-9278-d261e3e24111","Type":"ContainerStarted","Data":"059cbf6f2c1cf885645cdb09bee9f55a821e3b663291e0716e55b1ec14340630"}
Nov 02 18:10:04.446191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:04.445627    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t" event={"ID":"39fe532f-9e72-4297-9278-d261e3e24111","Type":"ContainerStarted","Data":"e91d866993669d9bdc857abc36af61dc683a2fbe4571f8436b24155bed717d90"}
Nov 02 18:10:04.469552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:04.469497    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t" podStartSLOduration=1.4694825040000001 podStartE2EDuration="1.469482504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:04.465232523 +0000 UTC m=+2217.044641909" watchObservedRunningTime="2026-11-02 18:10:04.469482504 +0000 UTC m=+2217.048891889"
Nov 02 18:10:05.466244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:05.460128    5042 generic.go:356] "Generic (PLEG): container finished" podID="39fe532f-9e72-4297-9278-d261e3e24111" containerID="059cbf6f2c1cf885645cdb09bee9f55a821e3b663291e0716e55b1ec14340630" exitCode=0
Nov 02 18:10:05.466244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:05.460428    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t" event={"ID":"39fe532f-9e72-4297-9278-d261e3e24111","Type":"ContainerDied","Data":"059cbf6f2c1cf885645cdb09bee9f55a821e3b663291e0716e55b1ec14340630"}
Nov 02 18:10:06.785683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.785644    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:06.866214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.864817    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca\") pod \"39fe532f-9e72-4297-9278-d261e3e24111\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") "
Nov 02 18:10:06.866214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.865175    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss\" (UniqueName: \"kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss\") pod \"39fe532f-9e72-4297-9278-d261e3e24111\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") "
Nov 02 18:10:06.866214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.865232    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca" pod "39fe532f-9e72-4297-9278-d261e3e24111" (UID: "39fe532f-9e72-4297-9278-d261e3e24111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:10:06.866214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.865245    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca\") pod \"39fe532f-9e72-4297-9278-d261e3e24111\" (UID: \"39fe532f-9e72-4297-9278-d261e3e24111\") "
Nov 02 18:10:06.866214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.866177    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:06.867867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.867737    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss" pod "39fe532f-9e72-4297-9278-d261e3e24111" (UID: "39fe532f-9e72-4297-9278-d261e3e24111"). InnerVolumeSpecName "kube-api-access-l7kss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:10:06.868242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.868217    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca" pod "39fe532f-9e72-4297-9278-d261e3e24111" (UID: "39fe532f-9e72-4297-9278-d261e3e24111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:10:06.968691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.968657    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kss\" (UniqueName: \"kubernetes.io/projected/39fe532f-9e72-4297-9278-d261e3e24111-kube-api-access-l7kss\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:06.968691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:06.968689    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39fe532f-9e72-4297-9278-d261e3e24111-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:07.485564 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:07.485530    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"
Nov 02 18:10:07.485564 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:07.485542    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t" event={"ID":"39fe532f-9e72-4297-9278-d261e3e24111","Type":"ContainerDied","Data":"e91d866993669d9bdc857abc36af61dc683a2fbe4571f8436b24155bed717d90"}
Nov 02 18:10:07.485809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:07.485593    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91d866993669d9bdc857abc36af61dc683a2fbe4571f8436b24155bed717d90"
Nov 02 18:10:09.407885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.407298    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"]
Nov 02 18:10:09.411864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.408599    5042 cpu_manager.go:414] "containerMap: removing container" podUID="39fe532f-9e72-4297-9278-d261e3e24111" containerName="revoke-keys"
Nov 02 18:10:09.411864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.408629    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39fe532f-9e72-4297-9278-d261e3e24111" containerName="revoke-keys"
Nov 02 18:10:09.411864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.409002    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="39fe532f-9e72-4297-9278-d261e3e24111" containerName="revoke-keys"
Nov 02 18:10:09.433861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.429243    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.528412    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"]
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.529763    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.529803    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59296\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.529856    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.529885    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.529957    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.530020    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.530232    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.530277    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.538435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.530343    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.631519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.631481    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.631695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.631659    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.631695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.631686    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59296\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.631786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.631719    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.631889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.631866    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.631972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.631956    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632037    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632156    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632278    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632313    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632335    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632579    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.632696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.632665    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.634524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.634501    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.634699 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.634682    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.640250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.640225    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.640893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.640630    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59296\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296\") pod \"maas-default-gateway-openshift-default-65c448b547-cmqs4\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:09.869250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:09.869211    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:10.234050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.233992    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:10:10.234201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.234167    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:10:10.234243 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.234213    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:10:10.253166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.253111    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"]
Nov 02 18:10:10.567012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.566917    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" event={"ID":"7a691c30-4a84-4c9d-af0f-8bf46481d9d8","Type":"ContainerStarted","Data":"8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3"}
Nov 02 18:10:10.567012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.566986    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" event={"ID":"7a691c30-4a84-4c9d-af0f-8bf46481d9d8","Type":"ContainerStarted","Data":"ea9e340de2b43ff3f73b1f9ceff7e689a4110d352857fb1ee82570bc906a3b24"}
Nov 02 18:10:10.592432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.592364    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podStartSLOduration=1.592345731 podStartE2EDuration="1.592345731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:10.587775464 +0000 UTC m=+2223.167184851" watchObservedRunningTime="2026-11-02 18:10:10.592345731 +0000 UTC m=+2223.171755118"
Nov 02 18:10:10.872982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:10.872084    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:11.871479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:11.871445    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:11.871926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:11.871502    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:12.869623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:12.869583    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:12.869816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:12.869637    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:13.733892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.726766    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"]
Nov 02 18:10:13.737180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.737116    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"]
Nov 02 18:10:13.737388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.737138    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:13.742111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.742076    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:10:13.742232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.742142    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:10:13.932865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.894381    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:13.943319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.903937    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:13.943684 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.943649    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:13.944077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.944049    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5jj\" (UniqueName: \"kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:13.944508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:13.944481    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.046623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.046537    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.046792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.046627    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.046792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.046700    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5jj\" (UniqueName: \"kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.047698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.047668    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.049476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.049449    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.055433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.055407    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5jj\" (UniqueName: \"kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj\") pod \"maas-api-revoke-sub-7fc384ee-644z9\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.056336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.056310    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:14.405905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.405869    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"]
Nov 02 18:10:14.409649 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:10:14.409611    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b33c59_006a_4c3f_953a_12bc953149e1.slice/crio-e9e2d27ba8e564a861b64e3fab44e921446bca9b26e12f03e489e1e348548ac6 WatchSource:0}: Error finding container e9e2d27ba8e564a861b64e3fab44e921446bca9b26e12f03e489e1e348548ac6: Status 404 returned error can't find the container with id e9e2d27ba8e564a861b64e3fab44e921446bca9b26e12f03e489e1e348548ac6
Nov 02 18:10:14.669099 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.668995    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9" event={"ID":"19b33c59-006a-4c3f-953a-12bc953149e1","Type":"ContainerStarted","Data":"e9e2d27ba8e564a861b64e3fab44e921446bca9b26e12f03e489e1e348548ac6"}
Nov 02 18:10:14.870370 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.870335    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:14.870781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:14.870389    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:15.848918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:15.839914    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9" event={"ID":"19b33c59-006a-4c3f-953a-12bc953149e1","Type":"ContainerStarted","Data":"fa00ea7366bcd1f5ed6a9e8e861a48cc715bffaf6bafa540c314e9590c260745"}
Nov 02 18:10:15.879925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:15.876299    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:15.879925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:15.876373    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:15.913865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:15.900754    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9" podStartSLOduration=2.900729952 podStartE2EDuration="2.900729952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:15.885132505 +0000 UTC m=+2228.464541892" watchObservedRunningTime="2026-11-02 18:10:15.900729952 +0000 UTC m=+2228.480139338"
Nov 02 18:10:16.882864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:16.882294    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:16.882864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:16.882358    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:16.916943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:16.916902    5042 generic.go:356] "Generic (PLEG): container finished" podID="19b33c59-006a-4c3f-953a-12bc953149e1" containerID="fa00ea7366bcd1f5ed6a9e8e861a48cc715bffaf6bafa540c314e9590c260745" exitCode=0
Nov 02 18:10:16.917445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:16.917420    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9" event={"ID":"19b33c59-006a-4c3f-953a-12bc953149e1","Type":"ContainerDied","Data":"fa00ea7366bcd1f5ed6a9e8e861a48cc715bffaf6bafa540c314e9590c260745"}
Nov 02 18:10:17.870136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:17.870090    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:17.870354 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:17.870166    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:18.150495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.150456    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:18.263833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.263790    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj\" (UniqueName: \"kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj\") pod \"19b33c59-006a-4c3f-953a-12bc953149e1\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") "
Nov 02 18:10:18.264038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.263926    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca\") pod \"19b33c59-006a-4c3f-953a-12bc953149e1\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") "
Nov 02 18:10:18.264384 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.264351    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca" pod "19b33c59-006a-4c3f-953a-12bc953149e1" (UID: "19b33c59-006a-4c3f-953a-12bc953149e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:10:18.264991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.264959    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca\") pod \"19b33c59-006a-4c3f-953a-12bc953149e1\" (UID: \"19b33c59-006a-4c3f-953a-12bc953149e1\") "
Nov 02 18:10:18.266097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.266073    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:18.266343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.266319    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca" pod "19b33c59-006a-4c3f-953a-12bc953149e1" (UID: "19b33c59-006a-4c3f-953a-12bc953149e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:10:18.267043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.267013    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj" pod "19b33c59-006a-4c3f-953a-12bc953149e1" (UID: "19b33c59-006a-4c3f-953a-12bc953149e1"). InnerVolumeSpecName "kube-api-access-jl5jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:10:18.368609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.368574    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl5jj\" (UniqueName: \"kubernetes.io/projected/19b33c59-006a-4c3f-953a-12bc953149e1-kube-api-access-jl5jj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:18.368609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.368608    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b33c59-006a-4c3f-953a-12bc953149e1-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:18.548451 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:10:18.548403    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:10:18.870529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.870429    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-cmqs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Nov 02 18:10:18.870529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.870483    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Nov 02 18:10:18.946175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.946130    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9" event={"ID":"19b33c59-006a-4c3f-953a-12bc953149e1","Type":"ContainerDied","Data":"e9e2d27ba8e564a861b64e3fab44e921446bca9b26e12f03e489e1e348548ac6"}
Nov 02 18:10:18.946175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.946166    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"
Nov 02 18:10:18.946407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:18.946218    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e2d27ba8e564a861b64e3fab44e921446bca9b26e12f03e489e1e348548ac6"
Nov 02 18:10:20.004554 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:20.004519    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:20.005520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:20.005492    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:20.006580 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:20.006552    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:10:41.021936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.021893    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"]
Nov 02 18:10:41.024108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.024074    5042 cpu_manager.go:414] "containerMap: removing container" podUID="19b33c59-006a-4c3f-953a-12bc953149e1" containerName="revoke-keys"
Nov 02 18:10:41.024334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.024317    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b33c59-006a-4c3f-953a-12bc953149e1" containerName="revoke-keys"
Nov 02 18:10:41.024777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.024758    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b33c59-006a-4c3f-953a-12bc953149e1" containerName="revoke-keys"
Nov 02 18:10:41.201863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.200262    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"]
Nov 02 18:10:41.201863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.200428    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.282015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.281928    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:10:41.282306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.282287    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:10:41.384679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.384633    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.384936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.384913    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdjp\" (UniqueName: \"kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.385015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.384957    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.486720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.486676    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.487188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.487160    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdjp\" (UniqueName: \"kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.487381 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.487362    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.487855 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.487814    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.487855 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.491912    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.487855 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.496070    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdjp\" (UniqueName: \"kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp\") pod \"maas-api-revoke-sub-3a39c3cb-sm95j\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:41.553075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:41.552989    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:42.501905 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:10:42.501828    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b4fb4_017a_4778_9ddd_92518d782dfb.slice/crio-90659908e5db8e771360baab70e47416904c7344477d8dfcbee581ef343e09ef WatchSource:0}: Error finding container 90659908e5db8e771360baab70e47416904c7344477d8dfcbee581ef343e09ef: Status 404 returned error can't find the container with id 90659908e5db8e771360baab70e47416904c7344477d8dfcbee581ef343e09ef
Nov 02 18:10:42.514371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:42.513371    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"]
Nov 02 18:10:42.786134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:42.786032    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j" event={"ID":"f89b4fb4-017a-4778-9ddd-92518d782dfb","Type":"ContainerStarted","Data":"90659908e5db8e771360baab70e47416904c7344477d8dfcbee581ef343e09ef"}
Nov 02 18:10:43.562359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.562324    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"]
Nov 02 18:10:43.567500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.567475    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.579358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.579323    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"]
Nov 02 18:10:43.654685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.654648    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.655070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.655043    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.655134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.655098    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-755df\" (UniqueName: \"kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.756537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.756490    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.756753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.756565    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-755df\" (UniqueName: \"kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.756753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.756654    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.757291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.757257    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.759147 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.759126    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.766147 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.766125    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-755df\" (UniqueName: \"kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df\") pod \"maas-api-revoke-sub-879ffccf-mvrwt\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:43.798707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.798673    5042 generic.go:356] "Generic (PLEG): container finished" podID="f89b4fb4-017a-4778-9ddd-92518d782dfb" containerID="6fc3978173001747aa4cf10ff06c46771289a003db5eb40cb172967719b69ac0" exitCode=0
Nov 02 18:10:43.798920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.798770    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j" event={"ID":"f89b4fb4-017a-4778-9ddd-92518d782dfb","Type":"ContainerDied","Data":"6fc3978173001747aa4cf10ff06c46771289a003db5eb40cb172967719b69ac0"}
Nov 02 18:10:43.910917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:43.901154    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:44.282375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:44.282341    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"]
Nov 02 18:10:44.283305 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:10:44.283272    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90b1766_ee2e_4fc9_9f9c_b2f64704585e.slice/crio-cd183d89d192f003b939c27d8d890744ae68c8acb819bd23417f92adf181c91d WatchSource:0}: Error finding container cd183d89d192f003b939c27d8d890744ae68c8acb819bd23417f92adf181c91d: Status 404 returned error can't find the container with id cd183d89d192f003b939c27d8d890744ae68c8acb819bd23417f92adf181c91d
Nov 02 18:10:44.809800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:44.809756    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt" event={"ID":"e90b1766-ee2e-4fc9-9f9c-b2f64704585e","Type":"ContainerStarted","Data":"45d91bc35f0c5ccf19dc7ce229ea0e71af46957f493dec8e7f9fc9e7d14883e2"}
Nov 02 18:10:44.810318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:44.809828    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt" event={"ID":"e90b1766-ee2e-4fc9-9f9c-b2f64704585e","Type":"ContainerStarted","Data":"cd183d89d192f003b939c27d8d890744ae68c8acb819bd23417f92adf181c91d"}
Nov 02 18:10:44.834109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:44.833644    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt" podStartSLOduration=1.833625734 podStartE2EDuration="1.833625734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:44.828309873 +0000 UTC m=+2257.407719258" watchObservedRunningTime="2026-11-02 18:10:44.833625734 +0000 UTC m=+2257.413035122"
Nov 02 18:10:45.077898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.077868    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:45.173222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.173131    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp\" (UniqueName: \"kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp\") pod \"f89b4fb4-017a-4778-9ddd-92518d782dfb\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") "
Nov 02 18:10:45.173222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.173204    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca\") pod \"f89b4fb4-017a-4778-9ddd-92518d782dfb\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") "
Nov 02 18:10:45.173493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.173466    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca\") pod \"f89b4fb4-017a-4778-9ddd-92518d782dfb\" (UID: \"f89b4fb4-017a-4778-9ddd-92518d782dfb\") "
Nov 02 18:10:45.173698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.173659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca" pod "f89b4fb4-017a-4778-9ddd-92518d782dfb" (UID: "f89b4fb4-017a-4778-9ddd-92518d782dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:10:45.174181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.174160    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:45.175911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.175883    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca" pod "f89b4fb4-017a-4778-9ddd-92518d782dfb" (UID: "f89b4fb4-017a-4778-9ddd-92518d782dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:10:45.176242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.176216    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp" pod "f89b4fb4-017a-4778-9ddd-92518d782dfb" (UID: "f89b4fb4-017a-4778-9ddd-92518d782dfb"). InnerVolumeSpecName "kube-api-access-sfdjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:10:45.276569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.276529    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89b4fb4-017a-4778-9ddd-92518d782dfb-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:45.276569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.276567    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdjp\" (UniqueName: \"kubernetes.io/projected/f89b4fb4-017a-4778-9ddd-92518d782dfb-kube-api-access-sfdjp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:45.936870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.936027    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j" event={"ID":"f89b4fb4-017a-4778-9ddd-92518d782dfb","Type":"ContainerDied","Data":"90659908e5db8e771360baab70e47416904c7344477d8dfcbee581ef343e09ef"}
Nov 02 18:10:45.936870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.936100    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90659908e5db8e771360baab70e47416904c7344477d8dfcbee581ef343e09ef"
Nov 02 18:10:45.936870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.936177    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"
Nov 02 18:10:45.960118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.944045    5042 generic.go:356] "Generic (PLEG): container finished" podID="e90b1766-ee2e-4fc9-9f9c-b2f64704585e" containerID="45d91bc35f0c5ccf19dc7ce229ea0e71af46957f493dec8e7f9fc9e7d14883e2" exitCode=0
Nov 02 18:10:45.960118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:45.944479    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt" event={"ID":"e90b1766-ee2e-4fc9-9f9c-b2f64704585e","Type":"ContainerDied","Data":"45d91bc35f0c5ccf19dc7ce229ea0e71af46957f493dec8e7f9fc9e7d14883e2"}
Nov 02 18:10:47.208417 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.208380    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:47.303476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.303389    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca\") pod \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") "
Nov 02 18:10:47.303662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.303504    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df\" (UniqueName: \"kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df\") pod \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") "
Nov 02 18:10:47.303662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.303642    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca\") pod \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\" (UID: \"e90b1766-ee2e-4fc9-9f9c-b2f64704585e\") "
Nov 02 18:10:47.303796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.303738    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca" pod "e90b1766-ee2e-4fc9-9f9c-b2f64704585e" (UID: "e90b1766-ee2e-4fc9-9f9c-b2f64704585e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:10:47.304192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.304171    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:47.305946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.305918    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df" pod "e90b1766-ee2e-4fc9-9f9c-b2f64704585e" (UID: "e90b1766-ee2e-4fc9-9f9c-b2f64704585e"). InnerVolumeSpecName "kube-api-access-755df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:10:47.306076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.305943    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca" pod "e90b1766-ee2e-4fc9-9f9c-b2f64704585e" (UID: "e90b1766-ee2e-4fc9-9f9c-b2f64704585e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:10:47.405779 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.405740    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:47.405779 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:47.405773    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-755df\" (UniqueName: \"kubernetes.io/projected/e90b1766-ee2e-4fc9-9f9c-b2f64704585e-kube-api-access-755df\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:10:48.027642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:48.027575    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt" event={"ID":"e90b1766-ee2e-4fc9-9f9c-b2f64704585e","Type":"ContainerDied","Data":"cd183d89d192f003b939c27d8d890744ae68c8acb819bd23417f92adf181c91d"}
Nov 02 18:10:48.027642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:48.027636    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"
Nov 02 18:10:48.027924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:48.027692    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd183d89d192f003b939c27d8d890744ae68c8acb819bd23417f92adf181c91d"
Nov 02 18:10:48.574271 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:10:48.574228    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:10:55.298010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.297968    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"]
Nov 02 18:10:55.299819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.299747    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f89b4fb4-017a-4778-9ddd-92518d782dfb" containerName="revoke-keys"
Nov 02 18:10:55.300023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.300005    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89b4fb4-017a-4778-9ddd-92518d782dfb" containerName="revoke-keys"
Nov 02 18:10:55.300222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.300210    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e90b1766-ee2e-4fc9-9f9c-b2f64704585e" containerName="revoke-keys"
Nov 02 18:10:55.300312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.300301    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90b1766-ee2e-4fc9-9f9c-b2f64704585e" containerName="revoke-keys"
Nov 02 18:10:55.300609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.300596    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90b1766-ee2e-4fc9-9f9c-b2f64704585e" containerName="revoke-keys"
Nov 02 18:10:55.300686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.300679    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89b4fb4-017a-4778-9ddd-92518d782dfb" containerName="revoke-keys"
Nov 02 18:10:55.308527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.308501    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.315159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.315130    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"]
Nov 02 18:10:55.315587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.315569    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:10:55.318788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.318435    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:10:55.508110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.498373    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vdhz\" (UniqueName: \"kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.508486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.508458    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.508664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.508644    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.611357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.611261    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vdhz\" (UniqueName: \"kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.611687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.611660    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.611876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.611832    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.613057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.613032    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.921556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.867826    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vdhz\" (UniqueName: \"kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.923977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.923946    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c547fc5c-r2xgv\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:55.936384 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:55.936349    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:10:56.424867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:56.424073    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"]
Nov 02 18:10:57.102965 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.092693    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"]
Nov 02 18:10:57.131862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.131149    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.171346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.166347    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"agc-cred-5df426b52b49\""
Nov 02 18:10:57.171346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.166468    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"openshift-service-ca.crt\""
Nov 02 18:10:57.171346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.166542    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"routing-overlay\""
Nov 02 18:10:57.171346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.166629    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"kube-root-ca.crt\""
Nov 02 18:10:57.171346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.166691    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bee8f6\""
Nov 02 18:10:57.174396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.174367    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bee8f6\"/\"payload-processing-external-model-e2e--24159a0a-dockercfg-lk82g\""
Nov 02 18:10:57.253902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.239306    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.253902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.239412    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.253902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.239625    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.253902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.239725    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvqz\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.365919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.361711    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvqz\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.365919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.361895    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.365919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.361983    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.365919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.362202    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.365919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.363172    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.392694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.389291    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.419124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.416275    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:10:57.438507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.433115    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvqz\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.445427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.445325    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"]
Nov 02 18:10:57.451764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.451689    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv" event={"ID":"b66d1b9f-352f-40fa-af5c-bc04393d9e69","Type":"ContainerStarted","Data":"db727d43048a67dc40d55c3cf16fa10353def31bd9e7648acc963723fded0c14"}
Nov 02 18:10:57.488940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.488738    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") " pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:57.718504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:57.718465    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:58.057086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.057056    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"]
Nov 02 18:10:58.058528 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:10:58.058495    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527e8c9f_f8a3_4a8e_bdfa_5f6ff14b93a1.slice/crio-6505db3e4ab93802fe246d0a5e4bafc6cbfbf27d26bd0d305f763a1485bd7de8 WatchSource:0}: Error finding container 6505db3e4ab93802fe246d0a5e4bafc6cbfbf27d26bd0d305f763a1485bd7de8: Status 404 returned error can't find the container with id 6505db3e4ab93802fe246d0a5e4bafc6cbfbf27d26bd0d305f763a1485bd7de8
Nov 02 18:10:58.465021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.464979    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv" event={"ID":"b66d1b9f-352f-40fa-af5c-bc04393d9e69","Type":"ContainerStarted","Data":"ff9a095f8345a46a2b655d2d57b75c3785edf6cd8c8c9d5ed0037d9d7354fc96"}
Nov 02 18:10:58.467991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.467814    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl" event={"ID":"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1","Type":"ContainerStarted","Data":"d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73"}
Nov 02 18:10:58.467991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.467936    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:10:58.467991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.467960    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl" event={"ID":"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1","Type":"ContainerStarted","Data":"6505db3e4ab93802fe246d0a5e4bafc6cbfbf27d26bd0d305f763a1485bd7de8"}
Nov 02 18:10:58.498650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.498198    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv" podStartSLOduration=3.498178276 podStartE2EDuration="3.498178276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:58.491228571 +0000 UTC m=+2271.070637958" watchObservedRunningTime="2026-11-02 18:10:58.498178276 +0000 UTC m=+2271.077587662"
Nov 02 18:10:58.514493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:58.514424    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl" podStartSLOduration=2.51440492 podStartE2EDuration="2.51440492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:58.510154023 +0000 UTC m=+2271.089563409" watchObservedRunningTime="2026-11-02 18:10:58.51440492 +0000 UTC m=+2271.093814303"
Nov 02 18:10:59.480236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:59.480199    5042 generic.go:356] "Generic (PLEG): container finished" podID="b66d1b9f-352f-40fa-af5c-bc04393d9e69" containerID="ff9a095f8345a46a2b655d2d57b75c3785edf6cd8c8c9d5ed0037d9d7354fc96" exitCode=0
Nov 02 18:10:59.480723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:10:59.480300    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv" event={"ID":"b66d1b9f-352f-40fa-af5c-bc04393d9e69","Type":"ContainerDied","Data":"ff9a095f8345a46a2b655d2d57b75c3785edf6cd8c8c9d5ed0037d9d7354fc96"}
Nov 02 18:11:00.728220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.728192    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:11:00.912436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.912342    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca\") pod \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") "
Nov 02 18:11:00.912436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.912426    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz\" (UniqueName: \"kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz\") pod \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") "
Nov 02 18:11:00.912666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.912590    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca\") pod \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\" (UID: \"b66d1b9f-352f-40fa-af5c-bc04393d9e69\") "
Nov 02 18:11:00.913049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.913011    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca" pod "b66d1b9f-352f-40fa-af5c-bc04393d9e69" (UID: "b66d1b9f-352f-40fa-af5c-bc04393d9e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:00.913808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.913781    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:00.914919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.914887    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca" pod "b66d1b9f-352f-40fa-af5c-bc04393d9e69" (UID: "b66d1b9f-352f-40fa-af5c-bc04393d9e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:00.914919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:00.914912    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz" pod "b66d1b9f-352f-40fa-af5c-bc04393d9e69" (UID: "b66d1b9f-352f-40fa-af5c-bc04393d9e69"). InnerVolumeSpecName "kube-api-access-5vdhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:01.015562 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:01.015524    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vdhz\" (UniqueName: \"kubernetes.io/projected/b66d1b9f-352f-40fa-af5c-bc04393d9e69-kube-api-access-5vdhz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:01.015562 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:01.015559    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66d1b9f-352f-40fa-af5c-bc04393d9e69-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:01.551577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:01.551546    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"
Nov 02 18:11:01.551760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:01.551580    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv" event={"ID":"b66d1b9f-352f-40fa-af5c-bc04393d9e69","Type":"ContainerDied","Data":"db727d43048a67dc40d55c3cf16fa10353def31bd9e7648acc963723fded0c14"}
Nov 02 18:11:01.551760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:01.551629    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db727d43048a67dc40d55c3cf16fa10353def31bd9e7648acc963723fded0c14"
Nov 02 18:11:09.485541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:09.485508    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:11:15.261866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:15.240651    5042 scope.go:122] "RemoveContainer" containerID="1458d65198b98e0e1f9faefc64cf49d205a5f0df71a4425ac5689ab8df8eebf6"
Nov 02 18:11:15.321122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:15.321095    5042 scope.go:122] "RemoveContainer" containerID="201fcc84ead064dda8ad18d7aeb39de538e706975b5b06782b0574d3185d3ef0"
Nov 02 18:11:15.404932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:15.404104    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"]
Nov 02 18:11:18.542307 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:11:18.542219    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:11:20.404960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:20.404912    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl" podUID="527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" containerName="payload-processing" containerID="cri-o://d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73" gracePeriod=25
Nov 02 18:11:20.976577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:20.976550    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:11:21.081457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.081343    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz\") pod \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") "
Nov 02 18:11:21.081457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.081454    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume\") pod \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") "
Nov 02 18:11:21.081725 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.081494    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume\") pod \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") "
Nov 02 18:11:21.081798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.081779    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials\") pod \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\" (UID: \"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1\") "
Nov 02 18:11:21.085697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.085658    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume" pod "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" (UID: "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:21.086123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.086092    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume" pod "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" (UID: "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:21.086511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.086487    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:21.086592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.086516    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:21.089997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.089683    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials" pod "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" (UID: "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:21.091064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.091041    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz" pod "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" (UID: "527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1"). InnerVolumeSpecName "kube-api-access-8mvqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:21.219903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.191573    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:21.219903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.191613    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvqz\" (UniqueName: \"kubernetes.io/projected/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1-kube-api-access-8mvqz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:11:21.242692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.240570    5042 generic.go:356] "Generic (PLEG): container finished" podID="527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" containerID="d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73" exitCode=0
Nov 02 18:11:21.242692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.240875    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"
Nov 02 18:11:21.242692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.241284    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl" event={"ID":"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1","Type":"ContainerDied","Data":"d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73"}
Nov 02 18:11:21.242692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.241418    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl" event={"ID":"527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1","Type":"ContainerDied","Data":"6505db3e4ab93802fe246d0a5e4bafc6cbfbf27d26bd0d305f763a1485bd7de8"}
Nov 02 18:11:21.242692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.241448    5042 scope.go:122] "RemoveContainer" containerID="d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73"
Nov 02 18:11:21.284199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.283249    5042 scope.go:122] "RemoveContainer" containerID="d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73"
Nov 02 18:11:21.284199 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:11:21.283754    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73\": container with ID starting with d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73 not found: ID does not exist" containerID="d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73"
Nov 02 18:11:21.284199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.283789    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73"} err="failed to get container status \"d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73\": rpc error: code = NotFound desc = could not find container \"d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73\": container with ID starting with d27e08ead9a8301df28facd948d2ca3fc02a1455256c06e89856f05caa0c2b73 not found: ID does not exist"
Nov 02 18:11:21.311889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.311852    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"]
Nov 02 18:11:21.319607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.319569    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bee8f6/payload-processing-external-model-e2e-extmod-b-bee8f6-758bcjjhl"]
Nov 02 18:11:21.549190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:21.549160    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" path="/var/lib/kubelet/pods/527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1/volumes"
Nov 02 18:11:23.473011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:23.472966    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:23.477989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:23.477952    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:29.033714 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:29.033669    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"]
Nov 02 18:11:29.043818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:29.043783    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc60380-p4dqp"]
Nov 02 18:11:29.545291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:29.545258    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c8cbc5-4476-46f0-ba0a-b2c1fce20a82" path="/var/lib/kubelet/pods/53c8cbc5-4476-46f0-ba0a-b2c1fce20a82/volumes"
Nov 02 18:11:37.161595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.161555    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"]
Nov 02 18:11:37.162801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.162778    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b66d1b9f-352f-40fa-af5c-bc04393d9e69" containerName="revoke-keys"
Nov 02 18:11:37.162911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.162805    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66d1b9f-352f-40fa-af5c-bc04393d9e69" containerName="revoke-keys"
Nov 02 18:11:37.162911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.162868    5042 cpu_manager.go:414] "containerMap: removing container" podUID="527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" containerName="payload-processing"
Nov 02 18:11:37.162911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.162876    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" containerName="payload-processing"
Nov 02 18:11:37.163180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.163164    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66d1b9f-352f-40fa-af5c-bc04393d9e69" containerName="revoke-keys"
Nov 02 18:11:37.163220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.163191    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="527e8c9f-f8a3-4a8e-bdfa-5f6ff14b93a1" containerName="payload-processing"
Nov 02 18:11:37.168790 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.168767    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.175216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.175191    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"]
Nov 02 18:11:37.209804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.209651    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-12ef34d6c6c7\""
Nov 02 18:11:37.209804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.209726    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2db162\""
Nov 02 18:11:37.210507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.209822    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"kube-root-ca.crt\""
Nov 02 18:11:37.210507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.209905    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"openshift-service-ca.crt\""
Nov 02 18:11:37.210507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.210123    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-ff45ed7b3970\""
Nov 02 18:11:37.210507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.210288    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"routing-overlay\""
Nov 02 18:11:37.210507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.210386    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"payload-processing-external-model-e2e--96b2e561-dockercfg-v9qrb\""
Nov 02 18:11:37.210507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.210416    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2db162\"/\"agc-cred-87625acbb868\""
Nov 02 18:11:37.219068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.219040    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.219236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.219093    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.219340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.219314    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vvn2\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.219554 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.219526    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.321561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.321517    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.321756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.321595    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.321756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.321734    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vvn2\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.321960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.321940    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.322506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.322474    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.322640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.322586    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.324728 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.324702    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.330419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.330396    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vvn2\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2\") pod \"payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") " pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.519080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.519045    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:37.865895 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:11:37.862795    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacc8ae3_f53c_4513_8b16_2b91dc00cdd3.slice/crio-4b8f049e128868f1e84d54a7df3b024ce6cfbb8af075b8531e1a86c8cc17ad1c WatchSource:0}: Error finding container 4b8f049e128868f1e84d54a7df3b024ce6cfbb8af075b8531e1a86c8cc17ad1c: Status 404 returned error can't find the container with id 4b8f049e128868f1e84d54a7df3b024ce6cfbb8af075b8531e1a86c8cc17ad1c
Nov 02 18:11:37.865895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:37.864667    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"]
Nov 02 18:11:38.882076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:38.882033    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx" event={"ID":"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3","Type":"ContainerStarted","Data":"af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11"}
Nov 02 18:11:38.882622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:38.882145    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx" event={"ID":"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3","Type":"ContainerStarted","Data":"4b8f049e128868f1e84d54a7df3b024ce6cfbb8af075b8531e1a86c8cc17ad1c"}
Nov 02 18:11:38.882622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:38.882173    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:38.901202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:38.901141    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx" podStartSLOduration=1.901126749 podStartE2EDuration="1.901126749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:38.899367862 +0000 UTC m=+2311.478777245" watchObservedRunningTime="2026-11-02 18:11:38.901126749 +0000 UTC m=+2311.480536175"
Nov 02 18:11:39.037130 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:39.036135    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"]
Nov 02 18:11:39.045334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:39.045297    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe3627d-k9pct"]
Nov 02 18:11:39.549482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:39.549446    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65937f44-72de-4086-a67d-65b8e01dd70d" path="/var/lib/kubelet/pods/65937f44-72de-4086-a67d-65b8e01dd70d/volumes"
Nov 02 18:11:48.549726 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:11:48.549685    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:11:49.897674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:49.897642    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:11:52.044486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:52.044452    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"]
Nov 02 18:11:52.050211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:52.050184    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"]
Nov 02 18:11:52.055006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:52.054986    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-948975e1-jdzvv"]
Nov 02 18:11:52.059559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:52.059539    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fbef6f2-w8x9n"]
Nov 02 18:11:53.548372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:53.548336    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22631999-8bb9-470a-88a3-5cebb1123954" path="/var/lib/kubelet/pods/22631999-8bb9-470a-88a3-5cebb1123954/volumes"
Nov 02 18:11:53.549094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:53.549072    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84daf268-b4be-482a-87cb-22e2b06055a7" path="/var/lib/kubelet/pods/84daf268-b4be-482a-87cb-22e2b06055a7/volumes"
Nov 02 18:11:58.027146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:58.027102    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"]
Nov 02 18:11:58.032459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:58.032431    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-373e7f75-w7hkp"]
Nov 02 18:11:59.033113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:59.033077    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"]
Nov 02 18:11:59.038215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:59.038186    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1694dc9-cwl6s"]
Nov 02 18:11:59.579140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:59.579097    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805d215c-170f-4e5c-8500-b1a0d5d069a2" path="/var/lib/kubelet/pods/805d215c-170f-4e5c-8500-b1a0d5d069a2/volumes"
Nov 02 18:11:59.580338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:11:59.580313    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc7f92f-59bf-4176-991b-2751eb5df590" path="/var/lib/kubelet/pods/dfc7f92f-59bf-4176-991b-2751eb5df590/volumes"
Nov 02 18:12:12.117860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:12.117187    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"]
Nov 02 18:12:12.135862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:12.134620    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cc40b61-4thlg"]
Nov 02 18:12:13.574864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:13.569320    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b86aa3-38d8-4a9c-8892-2dd81a992c29" path="/var/lib/kubelet/pods/e5b86aa3-38d8-4a9c-8892-2dd81a992c29/volumes"
Nov 02 18:12:18.634374 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:12:18.634333    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:12:23.033970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:23.033924    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"]
Nov 02 18:12:23.048176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:23.048130    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e396bc0-64rgg"]
Nov 02 18:12:23.547660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:23.547625    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b08b8bd-23cf-4ac8-96c6-e0e65d38a225" path="/var/lib/kubelet/pods/2b08b8bd-23cf-4ac8-96c6-e0e65d38a225/volumes"
Nov 02 18:12:25.277954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:25.277917    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"]
Nov 02 18:12:25.278386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:25.278179    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" containerID="cri-o://9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92" gracePeriod=30
Nov 02 18:12:29.249970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:29.249034    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:12:29.249970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:29.249097    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:12:30.664563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.664528    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 18:12:30.861598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.861535    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.861915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.861891    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.862117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.862087    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.862291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.862276    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.862430 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.862416    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.862656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.862628    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.862792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.862779    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.863254 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.863188    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.865614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.865590    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy\") pod \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\" (UID: \"4a3aadd4-5c54-4a81-ab70-8258d5e042ee\") "
Nov 02 18:12:30.865996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.864387    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:30.865996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.865400    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:30.865996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.865915    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:30.866633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.866352    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:30.867063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.867034    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.867162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.867072    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.867162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.867087    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.867162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.867102    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.867492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.867224    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:30.922882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.921004    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:30.922882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.921222    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "kube-api-access-qbv7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:30.922882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.921487    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:30.923543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.923242    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo" pod "4a3aadd4-5c54-4a81-ab70-8258d5e042ee" (UID: "4a3aadd4-5c54-4a81-ab70-8258d5e042ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:30.968996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.968963    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.968996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.968992    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbv7m\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-kube-api-access-qbv7m\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.968996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.969002    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.969253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.969011    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:30.969253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:30.969020    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a3aadd4-5c54-4a81-ab70-8258d5e042ee-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:31.000159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.000124    5042 generic.go:356] "Generic (PLEG): container finished" podID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerID="9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92" exitCode=0
Nov 02 18:12:31.000353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.000186    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"
Nov 02 18:12:31.000353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.000205    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" event={"ID":"4a3aadd4-5c54-4a81-ab70-8258d5e042ee","Type":"ContainerDied","Data":"9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92"}
Nov 02 18:12:31.000353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.000276    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj" event={"ID":"4a3aadd4-5c54-4a81-ab70-8258d5e042ee","Type":"ContainerDied","Data":"4eeb48a7449363467c3ece9e904d98d242ab1f6b6a37d46478d91d6b75b7dd35"}
Nov 02 18:12:31.000353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.000304    5042 scope.go:122] "RemoveContainer" containerID="9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92"
Nov 02 18:12:31.015398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.015377    5042 scope.go:122] "RemoveContainer" containerID="d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"
Nov 02 18:12:31.032077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.031971    5042 scope.go:122] "RemoveContainer" containerID="9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92"
Nov 02 18:12:31.032940 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:12:31.032903    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92\": container with ID starting with 9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92 not found: ID does not exist" containerID="9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92"
Nov 02 18:12:31.033652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.032953    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92"} err="failed to get container status \"9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92\": rpc error: code = NotFound desc = could not find container \"9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92\": container with ID starting with 9ea89812548e8a3a6e2b92759358904c4efb8d5ea81605a36253f339984d1a92 not found: ID does not exist"
Nov 02 18:12:31.033652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.032978    5042 scope.go:122] "RemoveContainer" containerID="d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"
Nov 02 18:12:31.033652 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:12:31.033406    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19\": container with ID starting with d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19 not found: ID does not exist" containerID="d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"
Nov 02 18:12:31.033652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.033432    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19"} err="failed to get container status \"d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19\": rpc error: code = NotFound desc = could not find container \"d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19\": container with ID starting with d1e9aaa233d3a093489367e921eea87748cef1fc58ee28aac91a94a833550b19 not found: ID does not exist"
Nov 02 18:12:31.033950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.033875    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"]
Nov 02 18:12:31.039280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.039256    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-9dwdj"]
Nov 02 18:12:31.548714 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:31.548678    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" path="/var/lib/kubelet/pods/4a3aadd4-5c54-4a81-ab70-8258d5e042ee/volumes"
Nov 02 18:12:38.376410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:38.376370    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"]
Nov 02 18:12:38.376888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:38.376654    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy" containerID="cri-o://68b4f358aa3e11b84dd66c474ab7a8af24b4c24749a0ddce2dccdc89ae8afb3a" gracePeriod=30
Nov 02 18:12:41.030058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:41.028935    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:12:41.107868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:41.100669    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:12:41.159859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:41.151205    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:12:41.165398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:41.164529    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:12:44.183345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:44.164054    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:12:44.929572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:44.927966    5042 generic.go:356] "Generic (PLEG): container finished" podID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerID="68b4f358aa3e11b84dd66c474ab7a8af24b4c24749a0ddce2dccdc89ae8afb3a" exitCode=0
Nov 02 18:12:44.929572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:44.928170    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" event={"ID":"9129ca9b-5439-40df-b2a6-a2bfbadc4740","Type":"ContainerDied","Data":"68b4f358aa3e11b84dd66c474ab7a8af24b4c24749a0ddce2dccdc89ae8afb3a"}
Nov 02 18:12:45.019951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:44.998314    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 18:12:45.120150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120104    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.120357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120186    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.120357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120213    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.120473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120389    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.120473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120429    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.122351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120565    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.122351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120623    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.122351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120695    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.122351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.120727    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs\") pod \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\" (UID: \"9129ca9b-5439-40df-b2a6-a2bfbadc4740\") "
Nov 02 18:12:45.122667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.122381    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:45.122667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.122640    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:45.124184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.122819    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:45.124184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.123520    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:45.124184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.123865    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:45.127203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.127176    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:45.127404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.127359    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "kube-api-access-n5nl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:45.127527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.127445    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:45.130573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.129802    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token" pod "9129ca9b-5439-40df-b2a6-a2bfbadc4740" (UID: "9129ca9b-5439-40df-b2a6-a2bfbadc4740"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223645    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223686    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223702    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nl2\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-kube-api-access-n5nl2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223717    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223730    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223745    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223760    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223773    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.224249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.223787    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9129ca9b-5439-40df-b2a6-a2bfbadc4740-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:45.940578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.940547    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"
Nov 02 18:12:45.940757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.940578    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg" event={"ID":"9129ca9b-5439-40df-b2a6-a2bfbadc4740","Type":"ContainerDied","Data":"a8001f208ca5f4211f306ccaa898ac0580b2b4e07530ef5126e626b7e16b9cd4"}
Nov 02 18:12:45.940757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.940641    5042 scope.go:122] "RemoveContainer" containerID="68b4f358aa3e11b84dd66c474ab7a8af24b4c24749a0ddce2dccdc89ae8afb3a"
Nov 02 18:12:45.968118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.968085    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"]
Nov 02 18:12:45.973327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:45.973296    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-dp2bg"]
Nov 02 18:12:47.551173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:47.551129    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" path="/var/lib/kubelet/pods/9129ca9b-5439-40df-b2a6-a2bfbadc4740/volumes"
Nov 02 18:12:48.585768 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:12:48.585733    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:12:51.068611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.068580    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"]
Nov 02 18:12:51.069392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069375    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy"
Nov 02 18:12:51.069441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069394    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy"
Nov 02 18:12:51.070190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069453    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy"
Nov 02 18:12:51.070190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069462    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy"
Nov 02 18:12:51.070190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069614    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9129ca9b-5439-40df-b2a6-a2bfbadc4740" containerName="istio-proxy"
Nov 02 18:12:51.070190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069629    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy"
Nov 02 18:12:51.070190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.069638    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy"
Nov 02 18:12:51.074955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.074934    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.086317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.085352    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:12:51.086317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.085469    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:12:51.088206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.088180    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"]
Nov 02 18:12:51.274168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.255785    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.275709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.275678    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx26\" (UniqueName: \"kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.276254 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.276224    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.378320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.378217    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.378492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.378379    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx26\" (UniqueName: \"kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.378634 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.378616    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.379021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.378993    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.381810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.381785    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.387724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.387691    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx26\" (UniqueName: \"kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26\") pod \"maas-api-revoke-sub-4ddbf560-zzmrk\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.397925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.397704    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:51.698424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:51.698388    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"]
Nov 02 18:12:51.699144 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:12:51.699108    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee644284_f14a_4385_bb1e_4173b49c904a.slice/crio-3b559b985c92c30c60c907a346060c82bd5121f1c58eeebb4ae6539780ec818b WatchSource:0}: Error finding container 3b559b985c92c30c60c907a346060c82bd5121f1c58eeebb4ae6539780ec818b: Status 404 returned error can't find the container with id 3b559b985c92c30c60c907a346060c82bd5121f1c58eeebb4ae6539780ec818b
Nov 02 18:12:52.045069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:52.045018    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk" event={"ID":"ee644284-f14a-4385-bb1e-4173b49c904a","Type":"ContainerStarted","Data":"3b559b985c92c30c60c907a346060c82bd5121f1c58eeebb4ae6539780ec818b"}
Nov 02 18:12:53.056284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:53.056253    5042 generic.go:356] "Generic (PLEG): container finished" podID="ee644284-f14a-4385-bb1e-4173b49c904a" containerID="537e480912df2a38b24af8d06eadffb9a31cea8d25e33f732b921784a8b986bf" exitCode=0
Nov 02 18:12:53.056726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:53.056342    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk" event={"ID":"ee644284-f14a-4385-bb1e-4173b49c904a","Type":"ContainerDied","Data":"537e480912df2a38b24af8d06eadffb9a31cea8d25e33f732b921784a8b986bf"}
Nov 02 18:12:54.334258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.334231    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:54.457524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.457489    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26\" (UniqueName: \"kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26\") pod \"ee644284-f14a-4385-bb1e-4173b49c904a\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") "
Nov 02 18:12:54.457751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.457637    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca\") pod \"ee644284-f14a-4385-bb1e-4173b49c904a\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") "
Nov 02 18:12:54.457751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.457733    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca\") pod \"ee644284-f14a-4385-bb1e-4173b49c904a\" (UID: \"ee644284-f14a-4385-bb1e-4173b49c904a\") "
Nov 02 18:12:54.458225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.458192    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca" pod "ee644284-f14a-4385-bb1e-4173b49c904a" (UID: "ee644284-f14a-4385-bb1e-4173b49c904a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:54.458618 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.458583    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:54.460036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.460009    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca" pod "ee644284-f14a-4385-bb1e-4173b49c904a" (UID: "ee644284-f14a-4385-bb1e-4173b49c904a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:54.460147 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.460088    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26" pod "ee644284-f14a-4385-bb1e-4173b49c904a" (UID: "ee644284-f14a-4385-bb1e-4173b49c904a"). InnerVolumeSpecName "kube-api-access-bhx26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:54.560528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.560488    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhx26\" (UniqueName: \"kubernetes.io/projected/ee644284-f14a-4385-bb1e-4173b49c904a-kube-api-access-bhx26\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:54.560528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:54.560522    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee644284-f14a-4385-bb1e-4173b49c904a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:12:55.114482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:55.114453    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"
Nov 02 18:12:55.114699 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:55.114490    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk" event={"ID":"ee644284-f14a-4385-bb1e-4173b49c904a","Type":"ContainerDied","Data":"3b559b985c92c30c60c907a346060c82bd5121f1c58eeebb4ae6539780ec818b"}
Nov 02 18:12:55.114699 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:12:55.114550    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b559b985c92c30c60c907a346060c82bd5121f1c58eeebb4ae6539780ec818b"
Nov 02 18:13:09.528936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:09.528903    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:13:09.532025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:09.532002    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:13:09.537070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:09.537046    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:13:09.539815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:09.539791    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:13:15.422884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.422591    5042 scope.go:122] "RemoveContainer" containerID="6241a0b9eb6d553cf4b3ac9b29318b3384e09c1ccd15ae6214c054a4ea831339"
Nov 02 18:13:15.446052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.445991    5042 scope.go:122] "RemoveContainer" containerID="a3b6af3577ffaadc76311f2edae9223be81b358c39e0042a29982e4bd3e1985e"
Nov 02 18:13:15.458777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.458753    5042 scope.go:122] "RemoveContainer" containerID="e58887657b4702de298ab3d6b59b765de516cffe2a6e86ad59c3e21c9de8dcbf"
Nov 02 18:13:15.473298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.473273    5042 scope.go:122] "RemoveContainer" containerID="bf9de72c8ce4fe83e594e270e869741d549b1ae1660e2c5537c011f2af690cfc"
Nov 02 18:13:15.489323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.489303    5042 scope.go:122] "RemoveContainer" containerID="ca5f48f108b6f0915fc58a5ba8a77bc89027373811d2b22c6833f1b7ab054e64"
Nov 02 18:13:15.505600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.505562    5042 scope.go:122] "RemoveContainer" containerID="8ae9f7329d7b0570d1887178ff3799d9cc5e30b7eb1a2b064a05b74d4feb7b87"
Nov 02 18:13:15.520319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:15.520278    5042 scope.go:122] "RemoveContainer" containerID="bb1c86d18f4c75d475ea9dbae8cb42aa56171ea611a811a485d01ba47603a981"
Nov 02 18:13:18.542917 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:13:18.542879    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:13:36.539151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.539115    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"]
Nov 02 18:13:36.540276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.540248    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ee644284-f14a-4385-bb1e-4173b49c904a" containerName="revoke-keys"
Nov 02 18:13:36.540276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.540276    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee644284-f14a-4385-bb1e-4173b49c904a" containerName="revoke-keys"
Nov 02 18:13:36.540460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.540372    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy"
Nov 02 18:13:36.540460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.540380    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3aadd4-5c54-4a81-ab70-8258d5e042ee" containerName="istio-proxy"
Nov 02 18:13:36.544071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.540563    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee644284-f14a-4385-bb1e-4173b49c904a" containerName="revoke-keys"
Nov 02 18:13:36.547857 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.547236    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.558902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.557449    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-78383f-serving-cert\""
Nov 02 18:13:36.558902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.557826    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"]
Nov 02 18:13:36.701868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.694980    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.701868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.695233    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.701868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.695485    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.701868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.695514    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhg4\" (UniqueName: \"kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.844574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.844480    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.844742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.844599    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.844742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.844618    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhg4\" (UniqueName: \"kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.844826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.844773    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.845382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.845348    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.847327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.847303    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.847436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.847336    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.853831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.853805    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhg4\" (UniqueName: \"kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4\") pod \"maas-api-e2e-inf-a-78383f-76b79875-5ggvp\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:36.872027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:36.871993    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:37.349856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:37.339133    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"]
Nov 02 18:13:37.349856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:37.348181    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:13:37.510374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:37.510335    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" event={"ID":"5a60e20b-67cf-435b-8385-5df6b10ab242","Type":"ContainerStarted","Data":"e36e058ac1fd17908aea883eee004862449e00ae8421fa6c5970317b16e087be"}
Nov 02 18:13:38.629865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:38.629350    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" event={"ID":"5a60e20b-67cf-435b-8385-5df6b10ab242","Type":"ContainerStarted","Data":"efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02"}
Nov 02 18:13:38.635875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:38.630710    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:38.695880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:38.688507    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" podStartSLOduration=2.106662296 podStartE2EDuration="2.68848181s" totalImagesPullingTime="581.819514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:38.653309632 +0000 UTC m=+2431.232719020" watchObservedRunningTime="2026-11-02 18:13:38.68848181 +0000 UTC m=+2431.267891196"
Nov 02 18:13:39.481043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:39.480984    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 18:13:39.482489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:39.482362    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:13:44.544002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.543958    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"]
Nov 02 18:13:44.549349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.549321    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.555649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.555619    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:13:44.557106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.556919    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:13:44.572516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.557738    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"]
Nov 02 18:13:44.644469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.644417    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmxvn\" (UniqueName: \"kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.644691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.644517    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.645164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.645133    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.747082 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.747045    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.747294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.747129    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmxvn\" (UniqueName: \"kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.747294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.747165    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.747944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.747922    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.749640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.749616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.755609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.755582    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmxvn\" (UniqueName: \"kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn\") pod \"maas-api-revoke-sub-66af28c3-7g5r5\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:44.870323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:44.870221    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:45.292008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:45.291961    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"]
Nov 02 18:13:45.294399 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:13:45.294351    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81af6bbf_e007_41bb_8e97_0ae620322b05.slice/crio-8e59bbfe1acdb7126ea7d92938c757d9e7048589f8c9f4013b911e24196e844c WatchSource:0}: Error finding container 8e59bbfe1acdb7126ea7d92938c757d9e7048589f8c9f4013b911e24196e844c: Status 404 returned error can't find the container with id 8e59bbfe1acdb7126ea7d92938c757d9e7048589f8c9f4013b911e24196e844c
Nov 02 18:13:45.725649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:45.725607    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:13:45.890966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:45.890928    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5" event={"ID":"81af6bbf-e007-41bb-8e97-0ae620322b05","Type":"ContainerStarted","Data":"8e59bbfe1acdb7126ea7d92938c757d9e7048589f8c9f4013b911e24196e844c"}
Nov 02 18:13:46.902743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:46.902694    5042 generic.go:356] "Generic (PLEG): container finished" podID="81af6bbf-e007-41bb-8e97-0ae620322b05" containerID="1f964f3001cf3f1fc49d5d8c29d44f30b5cdca8c44caf2f3f14369c250a08646" exitCode=0
Nov 02 18:13:46.903402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:46.902833    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5" event={"ID":"81af6bbf-e007-41bb-8e97-0ae620322b05","Type":"ContainerDied","Data":"1f964f3001cf3f1fc49d5d8c29d44f30b5cdca8c44caf2f3f14369c250a08646"}
Nov 02 18:13:48.216578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.216505    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:48.386705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.386663    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca\") pod \"81af6bbf-e007-41bb-8e97-0ae620322b05\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") "
Nov 02 18:13:48.387940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.386789    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca\") pod \"81af6bbf-e007-41bb-8e97-0ae620322b05\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") "
Nov 02 18:13:48.387940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.386817    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn\" (UniqueName: \"kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn\") pod \"81af6bbf-e007-41bb-8e97-0ae620322b05\" (UID: \"81af6bbf-e007-41bb-8e97-0ae620322b05\") "
Nov 02 18:13:48.387940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.387235    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca" pod "81af6bbf-e007-41bb-8e97-0ae620322b05" (UID: "81af6bbf-e007-41bb-8e97-0ae620322b05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:48.387940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.387685    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:13:48.390787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.390760    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca" pod "81af6bbf-e007-41bb-8e97-0ae620322b05" (UID: "81af6bbf-e007-41bb-8e97-0ae620322b05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:48.390923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.390808    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn" pod "81af6bbf-e007-41bb-8e97-0ae620322b05" (UID: "81af6bbf-e007-41bb-8e97-0ae620322b05"). InnerVolumeSpecName "kube-api-access-zmxvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:48.489509 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.489423    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81af6bbf-e007-41bb-8e97-0ae620322b05-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:13:48.489509 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.489453    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmxvn\" (UniqueName: \"kubernetes.io/projected/81af6bbf-e007-41bb-8e97-0ae620322b05-kube-api-access-zmxvn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:13:48.555438 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:13:48.555398    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:13:48.955159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.955115    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5" event={"ID":"81af6bbf-e007-41bb-8e97-0ae620322b05","Type":"ContainerDied","Data":"8e59bbfe1acdb7126ea7d92938c757d9e7048589f8c9f4013b911e24196e844c"}
Nov 02 18:13:48.955361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.955185    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e59bbfe1acdb7126ea7d92938c757d9e7048589f8c9f4013b911e24196e844c"
Nov 02 18:13:48.955361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:48.955185    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"
Nov 02 18:13:56.198822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.198780    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"]
Nov 02 18:13:56.202730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.202697    5042 cpu_manager.go:414] "containerMap: removing container" podUID="81af6bbf-e007-41bb-8e97-0ae620322b05" containerName="revoke-keys"
Nov 02 18:13:56.202730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.202731    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81af6bbf-e007-41bb-8e97-0ae620322b05" containerName="revoke-keys"
Nov 02 18:13:56.205465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.205438    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="81af6bbf-e007-41bb-8e97-0ae620322b05" containerName="revoke-keys"
Nov 02 18:13:56.216738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.216712    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"]
Nov 02 18:13:56.217438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.216898    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.223819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.223783    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:13:56.230791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.228473    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:13:56.342549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.342514    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dmdz\" (UniqueName: \"kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.342769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.342748    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.343452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.342934    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.445218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.445179    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.446659 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.445343    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dmdz\" (UniqueName: \"kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.446659 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.445487    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.446659 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.445911    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.452414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.452343    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.454759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.454730    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dmdz\" (UniqueName: \"kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz\") pod \"maas-api-revoke-sub-bbf89ab6-bsbnj\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.537680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.537641    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:56.968206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:56.968176    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"]
Nov 02 18:13:56.969723 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:13:56.969701    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd1a193_ecbe_4e96_9753_f32370513e46.slice/crio-a4da4e94de7d8d46e263f30915b6a7598912e486336789b613274e3be42721da WatchSource:0}: Error finding container a4da4e94de7d8d46e263f30915b6a7598912e486336789b613274e3be42721da: Status 404 returned error can't find the container with id a4da4e94de7d8d46e263f30915b6a7598912e486336789b613274e3be42721da
Nov 02 18:13:57.184905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:57.184868    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj" event={"ID":"add1a193-ecbe-4e96-9753-f32370513e46","Type":"ContainerStarted","Data":"a4da4e94de7d8d46e263f30915b6a7598912e486336789b613274e3be42721da"}
Nov 02 18:13:58.199430 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:58.199395    5042 generic.go:356] "Generic (PLEG): container finished" podID="add1a193-ecbe-4e96-9753-f32370513e46" containerID="befd5149bd532c63bbe001d78c1e0fd8885e8258091b5ebef639fdfa2e50ab66" exitCode=0
Nov 02 18:13:58.199877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:58.199471    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj" event={"ID":"add1a193-ecbe-4e96-9753-f32370513e46","Type":"ContainerDied","Data":"befd5149bd532c63bbe001d78c1e0fd8885e8258091b5ebef639fdfa2e50ab66"}
Nov 02 18:13:59.419322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.419282    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:13:59.527622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.527588    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca\") pod \"add1a193-ecbe-4e96-9753-f32370513e46\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") "
Nov 02 18:13:59.527865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.527656    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca\") pod \"add1a193-ecbe-4e96-9753-f32370513e46\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") "
Nov 02 18:13:59.528765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.527992    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz\" (UniqueName: \"kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz\") pod \"add1a193-ecbe-4e96-9753-f32370513e46\" (UID: \"add1a193-ecbe-4e96-9753-f32370513e46\") "
Nov 02 18:13:59.528765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.528056    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca" pod "add1a193-ecbe-4e96-9753-f32370513e46" (UID: "add1a193-ecbe-4e96-9753-f32370513e46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:59.528765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.528759    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:13:59.530545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.530494    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca" pod "add1a193-ecbe-4e96-9753-f32370513e46" (UID: "add1a193-ecbe-4e96-9753-f32370513e46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:59.530742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.530719    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz" pod "add1a193-ecbe-4e96-9753-f32370513e46" (UID: "add1a193-ecbe-4e96-9753-f32370513e46"). InnerVolumeSpecName "kube-api-access-8dmdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:59.630383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.630301    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dmdz\" (UniqueName: \"kubernetes.io/projected/add1a193-ecbe-4e96-9753-f32370513e46-kube-api-access-8dmdz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:13:59.630383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:13:59.630339    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/add1a193-ecbe-4e96-9753-f32370513e46-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:00.278527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:00.276123    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj" event={"ID":"add1a193-ecbe-4e96-9753-f32370513e46","Type":"ContainerDied","Data":"a4da4e94de7d8d46e263f30915b6a7598912e486336789b613274e3be42721da"}
Nov 02 18:14:00.278527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:00.276191    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4da4e94de7d8d46e263f30915b6a7598912e486336789b613274e3be42721da"
Nov 02 18:14:00.278527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:00.276266    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"
Nov 02 18:14:06.035003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:06.034941    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"]
Nov 02 18:14:06.040459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:06.040424    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25386faa-rfm9f"]
Nov 02 18:14:07.626155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:07.626109    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283efc9e-0afe-4e39-8674-e2943a39e96d" path="/var/lib/kubelet/pods/283efc9e-0afe-4e39-8674-e2943a39e96d/volumes"
Nov 02 18:14:08.028173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:08.028126    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"]
Nov 02 18:14:08.033652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:08.033614    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee20b4b6-xrfp7"]
Nov 02 18:14:09.545824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:09.545782    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911da677-275c-4c93-b399-90a2a4098f54" path="/var/lib/kubelet/pods/911da677-275c-4c93-b399-90a2a4098f54/volumes"
Nov 02 18:14:13.863000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:13.154956    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:13.176668    5042 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-98bw2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:13.864111    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-98bw2" podUID="6377c122-4ad5-4829-a2df-ed251a70d3bc" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.537215    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.245:50052\" within 1s: context deadline exceeded"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:14:14.537404    5042 logging.go:55] [core] [Channel #2818 SubChannel #2822]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.245:50052", ServerName: "10.132.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.245:50052: operation was canceled"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.557830    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.242:50052\" within 1s: context deadline exceeded"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:14:14.557997    5042 logging.go:55] [core] [Channel #2816 SubChannel #2824]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.242:50052", ServerName: "10.132.0.242:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.242:50052: operation was canceled"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.587002    5042 patch_prober.go:29] interesting pod/router-default-c7d5754d7-v6znk container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.15:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.587100    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-c7d5754d7-v6znk" podUID="7adb34aa-6a54-4a58-8761-49b5de183bba" containerName="router" probeResult="failure" output="Get \"http://10.132.0.15:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.798308    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" podUID="588a37ce-4b38-4224-9f67-adb7273f76b5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.863980    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:14.864211    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:15.798171    5042 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-vktlz container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:15.798261    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz" podUID="fa18f4d4-d9fa-4802-b90b-e7044d222b39" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:15.841018    5042 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-98bw2 container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:15.841085    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-98bw2" podUID="6377c122-4ad5-4829-a2df-ed251a70d3bc" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:15.859186    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-lx6js" podUID="06bdf526-bccd-4303-b9f5-ff3131a4ae96" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.37:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:15.864239    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:16.607667    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:16.654007    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-789486f7c8-sx6g4" podUID="2e8003ad-c2d2-483b-8ccf-d48aadd55aed" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.36:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:16.753982    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ljs7q" podUID="5488c1fe-6a3d-464a-bd15-2377b7740b9b" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.39:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:17.391819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:16.754022    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-k9mfj" podUID="d56437ee-40bb-439d-812f-1a55863eac69" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.38:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.495251    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh" podUID="23fa0458-aa25-47cf-9b62-00385b8646fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.497418    5042 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-vktlz container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.497479    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-vktlz" podUID="fa18f4d4-d9fa-4802-b90b-e7044d222b39" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.28:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.502140    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.542424    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.556769    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.193:50052\" within 1s: context deadline exceeded"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.591987    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-c8zlh" podUID="23fa0458-aa25-47cf-9b62-00385b8646fd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.604749    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9" podUID="6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.43:8081/readyz\": dial tcp 10.132.0.43:8081: connect: connection refused"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.618942    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"]
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.618981    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"]
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.619204    5042 patch_prober.go:29] interesting pod/image-registry-5bb6c9c46c-66dr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.30:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.619344    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bb6c9c46c-66dr9" podUID="417a9298-1e3c-46bb-965c-aab3c75b8728" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.30:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.637318    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53691d9-bf9m4"]
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.645254    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e34f552-bdslm"]
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:14:28.691406    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e4902_3ee4_40f1_a1d5_a7720cf9d7a0.slice/crio-conmon-b77302fcba3cafa3033a0760926bce7a495265cc1900cad601737f16f09707da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:14:28.691429    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e4902_3ee4_40f1_a1d5_a7720cf9d7a0.slice/crio-conmon-b77302fcba3cafa3033a0760926bce7a495265cc1900cad601737f16f09707da.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:14:29.532920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:28.740452    5042 scope.go:122] "RemoveContainer" containerID="04df5d9667d9cca6b8d4ec537deb8cb61fd246de06a6b4fa91c7e9773694f0db"
Nov 02 18:14:29.658388 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:14:29.641740    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:14:29.658388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:29.652319    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd21099-9f25-426d-8b01-22d2547ec98a" path="/var/lib/kubelet/pods/cdd21099-9f25-426d-8b01-22d2547ec98a/volumes"
Nov 02 18:14:29.660799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:29.660768    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d445901d-dd96-4bdf-9bc1-c95bb7c7b301" path="/var/lib/kubelet/pods/d445901d-dd96-4bdf-9bc1-c95bb7c7b301/volumes"
Nov 02 18:14:30.024819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:30.024777    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9" event={"ID":"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0","Type":"ContainerDied","Data":"b77302fcba3cafa3033a0760926bce7a495265cc1900cad601737f16f09707da"}
Nov 02 18:14:30.025357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:30.025336    5042 scope.go:122] "RemoveContainer" containerID="b77302fcba3cafa3033a0760926bce7a495265cc1900cad601737f16f09707da"
Nov 02 18:14:30.033035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:30.032996    5042 generic.go:356] "Generic (PLEG): container finished" podID="6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0" containerID="b77302fcba3cafa3033a0760926bce7a495265cc1900cad601737f16f09707da" exitCode=1
Nov 02 18:14:30.822772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:30.822728    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 18:14:31.047449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:31.047413    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9" event={"ID":"6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0","Type":"ContainerStarted","Data":"a1646872d02a1314c6f61bcb6ff55c208c12b8aefb666ec0cef93129423065a0"}
Nov 02 18:14:31.047665 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:31.047647    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 18:14:35.096306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.096259    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"]
Nov 02 18:14:35.122038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.122002    5042 cpu_manager.go:414] "containerMap: removing container" podUID="add1a193-ecbe-4e96-9753-f32370513e46" containerName="revoke-keys"
Nov 02 18:14:35.122322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.122297    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add1a193-ecbe-4e96-9753-f32370513e46" containerName="revoke-keys"
Nov 02 18:14:35.122739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.122696    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="add1a193-ecbe-4e96-9753-f32370513e46" containerName="revoke-keys"
Nov 02 18:14:35.127933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.127909    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.134779 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.134748    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:14:35.134779 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.134750    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:14:35.142809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.142296    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"]
Nov 02 18:14:35.156622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.156583    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdbhd\" (UniqueName: \"kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.157096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.157069    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.158739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.157233    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.259053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.259016    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdbhd\" (UniqueName: \"kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.259247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.259153    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.259247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.259190    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.262140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.262118    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.279252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.279111    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdbhd\" (UniqueName: \"kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.280662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.280617    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85b8b498-b5xz2\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:35.473601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:35.473567    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:36.071305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:36.071270    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"]
Nov 02 18:14:36.108769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:36.108728    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2" event={"ID":"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3","Type":"ContainerStarted","Data":"0442e3876737ce4592493f3f8a73966d7177fe7a50ea7ef8177bac7e5b197a6e"}
Nov 02 18:14:37.122881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:37.122799    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2" event={"ID":"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3","Type":"ContainerStarted","Data":"62fc17b5ddfcd747d002f63936c203d1251e0cd647b50ee4a3306325d8fea2b7"}
Nov 02 18:14:37.141616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:37.141561    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2" podStartSLOduration=2.141545408 podStartE2EDuration="2.141545408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:37.140061542 +0000 UTC m=+2489.719470929" watchObservedRunningTime="2026-11-02 18:14:37.141545408 +0000 UTC m=+2489.720954796"
Nov 02 18:14:38.133561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:38.133526    5042 generic.go:356] "Generic (PLEG): container finished" podID="0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" containerID="62fc17b5ddfcd747d002f63936c203d1251e0cd647b50ee4a3306325d8fea2b7" exitCode=0
Nov 02 18:14:38.134918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:38.133624    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2" event={"ID":"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3","Type":"ContainerDied","Data":"62fc17b5ddfcd747d002f63936c203d1251e0cd647b50ee4a3306325d8fea2b7"}
Nov 02 18:14:39.388040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.388011    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:39.424208 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.424114    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca\") pod \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") "
Nov 02 18:14:39.424208 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.424180    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca\") pod \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") "
Nov 02 18:14:39.424411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.424395    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd\" (UniqueName: \"kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd\") pod \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\" (UID: \"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3\") "
Nov 02 18:14:39.424642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.424535    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca" pod "0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" (UID: "0610f7f6-b2f5-4c32-ac73-1a79a5d036f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:14:39.425431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.425408    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:39.430644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.430612    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca" pod "0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" (UID: "0610f7f6-b2f5-4c32-ac73-1a79a5d036f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:14:39.431235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.430696    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd" pod "0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" (UID: "0610f7f6-b2f5-4c32-ac73-1a79a5d036f3"). InnerVolumeSpecName "kube-api-access-hdbhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:39.527020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.526982    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:39.527020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:39.527011    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdbhd\" (UniqueName: \"kubernetes.io/projected/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3-kube-api-access-hdbhd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:40.154378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:40.154346    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"
Nov 02 18:14:40.154591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:40.154374    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2" event={"ID":"0610f7f6-b2f5-4c32-ac73-1a79a5d036f3","Type":"ContainerDied","Data":"0442e3876737ce4592493f3f8a73966d7177fe7a50ea7ef8177bac7e5b197a6e"}
Nov 02 18:14:40.154591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:40.154430    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0442e3876737ce4592493f3f8a73966d7177fe7a50ea7ef8177bac7e5b197a6e"
Nov 02 18:14:41.044474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:41.044426    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"]
Nov 02 18:14:41.046297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:41.046263    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d46fb37-j8s7m"]
Nov 02 18:14:41.545792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:41.545757    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6355fee8-e959-48dd-9b99-c1a9d9617ddd" path="/var/lib/kubelet/pods/6355fee8-e959-48dd-9b99-c1a9d9617ddd/volumes"
Nov 02 18:14:45.631740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.631705    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"]
Nov 02 18:14:45.633004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.632981    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" containerName="revoke-keys"
Nov 02 18:14:45.633082 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.633006    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" containerName="revoke-keys"
Nov 02 18:14:45.633204 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.633191    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" containerName="revoke-keys"
Nov 02 18:14:45.663406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.662376    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"]
Nov 02 18:14:45.663406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.662526    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.672283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.672254    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:14:45.672456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.672376    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:14:45.793994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.793948    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljlz\" (UniqueName: \"kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.794271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.794234    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.794463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.794437    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.931532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.931433    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.932368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.932133    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljlz\" (UniqueName: \"kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.932489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.932470    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.970045 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.970009    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.975427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.975235    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:45.975606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:45.975495    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljlz\" (UniqueName: \"kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz\") pod \"maas-api-revoke-sub-015bf5e2-49fgk\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:46.025863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:46.004192    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:46.684762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:46.684732    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"]
Nov 02 18:14:46.688036 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:14:46.687987    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3078b6e0_f97d_4fb0_88c9_6d8f07b048a3.slice/crio-242579127bd1d65b51dd976a1d1b1d361cf80919749916fec295bd04fa6be12d WatchSource:0}: Error finding container 242579127bd1d65b51dd976a1d1b1d361cf80919749916fec295bd04fa6be12d: Status 404 returned error can't find the container with id 242579127bd1d65b51dd976a1d1b1d361cf80919749916fec295bd04fa6be12d
Nov 02 18:14:47.394638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:47.394597    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk" event={"ID":"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3","Type":"ContainerStarted","Data":"d15ce962118e32a9dd7b3352e365b64db88df8650d7534beb18e1db9c138af48"}
Nov 02 18:14:47.394830 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:47.394662    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk" event={"ID":"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3","Type":"ContainerStarted","Data":"242579127bd1d65b51dd976a1d1b1d361cf80919749916fec295bd04fa6be12d"}
Nov 02 18:14:47.418822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:47.418755    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk" podStartSLOduration=2.418735825 podStartE2EDuration="2.418735825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:47.410876412 +0000 UTC m=+2499.990285799" watchObservedRunningTime="2026-11-02 18:14:47.418735825 +0000 UTC m=+2499.998145212"
Nov 02 18:14:48.406785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:48.406683    5042 generic.go:356] "Generic (PLEG): container finished" podID="3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" containerID="d15ce962118e32a9dd7b3352e365b64db88df8650d7534beb18e1db9c138af48" exitCode=0
Nov 02 18:14:48.406785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:48.406762    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk" event={"ID":"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3","Type":"ContainerDied","Data":"d15ce962118e32a9dd7b3352e365b64db88df8650d7534beb18e1db9c138af48"}
Nov 02 18:14:48.608805 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:14:48.608774    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:14:49.664035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.664002    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:49.776114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.776071    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca\") pod \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") "
Nov 02 18:14:49.776317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.776141    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca\") pod \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") "
Nov 02 18:14:49.777180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.776312    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz\" (UniqueName: \"kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz\") pod \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\" (UID: \"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3\") "
Nov 02 18:14:49.777180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.776487    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca" pod "3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" (UID: "3078b6e0-f97d-4fb0-88c9-6d8f07b048a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:14:49.778212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.777629    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:49.781044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.780986    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz" pod "3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" (UID: "3078b6e0-f97d-4fb0-88c9-6d8f07b048a3"). InnerVolumeSpecName "kube-api-access-vljlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:49.781437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.781412    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca" pod "3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" (UID: "3078b6e0-f97d-4fb0-88c9-6d8f07b048a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:14:49.879113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.879078    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:49.879113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:49.879109    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vljlz\" (UniqueName: \"kubernetes.io/projected/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3-kube-api-access-vljlz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:14:50.519895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:50.519788    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk" event={"ID":"3078b6e0-f97d-4fb0-88c9-6d8f07b048a3","Type":"ContainerDied","Data":"242579127bd1d65b51dd976a1d1b1d361cf80919749916fec295bd04fa6be12d"}
Nov 02 18:14:50.520110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:50.519949    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242579127bd1d65b51dd976a1d1b1d361cf80919749916fec295bd04fa6be12d"
Nov 02 18:14:50.520110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:50.520059    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"
Nov 02 18:14:52.030151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:52.030117    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"]
Nov 02 18:14:52.035387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:52.035360    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adb0f52c-22nnm"]
Nov 02 18:14:52.060442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:52.060400    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-tfbt9"
Nov 02 18:14:53.596992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:14:53.596958    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f35cdd5-7ada-4333-957e-a301964fbc49" path="/var/lib/kubelet/pods/5f35cdd5-7ada-4333-957e-a301964fbc49/volumes"
Nov 02 18:15:00.144020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.143953    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"]
Nov 02 18:15:00.145278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.145251    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" containerName="revoke-keys"
Nov 02 18:15:00.145278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.145278    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" containerName="revoke-keys"
Nov 02 18:15:00.145567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.145549    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" containerName="revoke-keys"
Nov 02 18:15:00.149582 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.149559    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.152361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.152336    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"]
Nov 02 18:15:00.158347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.158324    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.160547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.160526    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"]
Nov 02 18:15:00.161731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.161711    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:15:00.265233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.265196    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.299003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.285857    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"]
Nov 02 18:15:00.299003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.297393    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"]
Nov 02 18:15:00.299003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.297759    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.308702    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.308753    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.311595    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzgc5\" (UniqueName: \"kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5\") pod \"maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.311650    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4wf\" (UniqueName: \"kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf\") pod \"maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.320359    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"]
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.320404    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"]
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.320431    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"]
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.320456    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"]
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.320483    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"]
Nov 02 18:15:00.324921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.321533    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.333121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.329051    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"]
Nov 02 18:15:00.333121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.332175    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.334950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.334929    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"]
Nov 02 18:15:00.335149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.335130    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.339255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.339234    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"]
Nov 02 18:15:00.340051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.340030    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.343802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.343783    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"]
Nov 02 18:15:00.343930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.343906    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.348208 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.348184    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"]
Nov 02 18:15:00.356432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.356406    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"]
Nov 02 18:15:00.366994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.365799    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"]
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.445476    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.445577    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.445718    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsmtv\" (UniqueName: \"kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv\") pod \"maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.445782    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.445864    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwbz\" (UniqueName: \"kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.445972    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzgc5\" (UniqueName: \"kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5\") pod \"maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446028    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4wf\" (UniqueName: \"kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf\") pod \"maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446105    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446195    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446256    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w29c7\" (UniqueName: \"kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446294    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8z5v\" (UniqueName: \"kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446331    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446425    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smhb\" (UniqueName: \"kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb\") pod \"maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446455    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446484    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446523    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446602    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mschr\" (UniqueName: \"kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.446655    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.448770    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.450210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.449537    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.504858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.472413    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzgc5\" (UniqueName: \"kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5\") pod \"maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.504858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.474896    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4wf\" (UniqueName: \"kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf\") pod \"maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.552139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.552064    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"]
Nov 02 18:15:00.553344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553311    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.553519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553415    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.553519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553468    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w29c7\" (UniqueName: \"kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.553519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553500    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8z5v\" (UniqueName: \"kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.553703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553532    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.553703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553611    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smhb\" (UniqueName: \"kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb\") pod \"maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.553703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553640    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553718    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mschr\" (UniqueName: \"kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553760    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553796    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.553889    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.554001    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsmtv\" (UniqueName: \"kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv\") pod \"maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.554058    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.556065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.554108    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwbz\" (UniqueName: \"kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.561177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.559156    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.561177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.559925    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.561177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.560918    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.562483    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.564655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.564296    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.566446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.566003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwbz\" (UniqueName: \"kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.569423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.569390    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w29c7\" (UniqueName: \"kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.570136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.570117    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.571994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.571961    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:00.572479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.572459    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.578632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.573912    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8z5v\" (UniqueName: \"kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.578632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.576612    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:00.578632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.576743    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:00.581164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.580439    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br\") pod \"maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.581164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.580962    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smhb\" (UniqueName: \"kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb\") pod \"maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.582745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.581907    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"]
Nov 02 18:15:00.584948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.582996    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:00.591538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.591506    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mschr\" (UniqueName: \"kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr\") pod \"maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.593182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.593159    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"]
Nov 02 18:15:00.597128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.597106    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:00.612673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.607960    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:00.623976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.621823    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsmtv\" (UniqueName: \"kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv\") pod \"maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:00.626486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.626140    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:00.629713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.629642    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"]
Nov 02 18:15:00.638833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.638805    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"]
Nov 02 18:15:00.660242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.658582    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"]
Nov 02 18:15:00.672415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.672393    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"]
Nov 02 18:15:00.702578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.702462    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:00.711888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.711701    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:00.713687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.712722    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:00.726193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.724057    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:00.772375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.772326    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-sjzz6\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:00.772577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.772482    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtd54\" (UniqueName: \"kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:00.773110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.773063    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9cv\" (UniqueName: \"kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:00.773266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.773151    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:00.773336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.773293    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2q67\" (UniqueName: \"kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67\") pod \"maas-api-key-cleanup-29894055-sjzz6\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:00.773397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.773371    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:00.898748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.898697    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.947938    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.948028    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-sjzz6\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.948166    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtd54\" (UniqueName: \"kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.948420    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9cv\" (UniqueName: \"kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.948501    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.948654    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2q67\" (UniqueName: \"kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67\") pod \"maas-api-key-cleanup-29894055-sjzz6\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.949796    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.953033    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.958261    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-sjzz6\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:01.020923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:00.963202    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2q67\" (UniqueName: \"kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67\") pod \"maas-api-key-cleanup-29894055-sjzz6\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:01.050909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.047922    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:01.050909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.050571    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtd54\" (UniqueName: \"kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:01.060071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.057277    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9cv\" (UniqueName: \"kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv\") pod \"maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:01.377863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.371395    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:01.413103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.382952    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:01.914224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.914177    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"]
Nov 02 18:15:01.990645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.990588    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"]
Nov 02 18:15:01.994513 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:01.994467    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7b295a_1204_4db7_81dd_bb7483eff02d.slice/crio-0019f8588eb7be7a82fba4063f30ae636b409708278b63907c35181369e0b700 WatchSource:0}: Error finding container 0019f8588eb7be7a82fba4063f30ae636b409708278b63907c35181369e0b700: Status 404 returned error can't find the container with id 0019f8588eb7be7a82fba4063f30ae636b409708278b63907c35181369e0b700
Nov 02 18:15:01.996709 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:01.996667    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeecd43d_55b7_4310_8d29_3d7db1d92847.slice/crio-db707eb03ff5e363290bd326e3ae1fda176c45d3d0bfeb90fb78a1ea86e73536 WatchSource:0}: Error finding container db707eb03ff5e363290bd326e3ae1fda176c45d3d0bfeb90fb78a1ea86e73536: Status 404 returned error can't find the container with id db707eb03ff5e363290bd326e3ae1fda176c45d3d0bfeb90fb78a1ea86e73536
Nov 02 18:15:02.000287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:01.999601    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"]
Nov 02 18:15:02.039758 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:02.039725    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9eff2e_1e2a_4472_b382_46a8fa8a995e.slice/crio-a380acccfcfe11f1175eb1966ddcb4b5c33aac224b04f25df454f8cfd4b8c5e9 WatchSource:0}: Error finding container a380acccfcfe11f1175eb1966ddcb4b5c33aac224b04f25df454f8cfd4b8c5e9: Status 404 returned error can't find the container with id a380acccfcfe11f1175eb1966ddcb4b5c33aac224b04f25df454f8cfd4b8c5e9
Nov 02 18:15:02.060184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.060105    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"]
Nov 02 18:15:02.092801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.092776    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"]
Nov 02 18:15:02.131974 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.131946    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"]
Nov 02 18:15:02.153395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.153366    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"]
Nov 02 18:15:02.544791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.543960    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"]
Nov 02 18:15:02.773397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.766446    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"]
Nov 02 18:15:02.781706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.781666    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"]
Nov 02 18:15:02.794711 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.794664    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"]
Nov 02 18:15:02.865193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.865150    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy" containerID="cri-o://9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74" gracePeriod=30
Nov 02 18:15:02.982136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.968168    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"]
Nov 02 18:15:02.982136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:02.968276    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerStarted","Data":"b15a6dc6e2e305f8576ca11a5559ecab8d1bcbcdfb3d6823bf9d392fb33f958f"}
Nov 02 18:15:03.054698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.054000    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerStarted","Data":"0019f8588eb7be7a82fba4063f30ae636b409708278b63907c35181369e0b700"}
Nov 02 18:15:03.057891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.057829    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerStarted","Data":"a380acccfcfe11f1175eb1966ddcb4b5c33aac224b04f25df454f8cfd4b8c5e9"}
Nov 02 18:15:03.060111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.060054    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerStarted","Data":"10fe12b1003229239c5adcce9c1785af09409ce211eb202eadedc35ff897677a"}
Nov 02 18:15:03.062718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.062676    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerStarted","Data":"7cc29d86bddedc45ab4ecb70203ad79dbdc3a03b73d9f0d668735ee67fd74b0c"}
Nov 02 18:15:03.085358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.085313    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" event={"ID":"8431f13a-72b7-418f-8563-10812bf2c419","Type":"ContainerStarted","Data":"3f5a33a7b35f4b06fd9825d2df917732f46c34963f688ded222c6b7f5abd4407"}
Nov 02 18:15:03.091942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.091895    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" event={"ID":"ce6bc700-de9b-4e95-8d3a-a704283cd4fc","Type":"ContainerStarted","Data":"72dde2415fb8ecc4c7669470d08f768580a8675b4f6c9dcdeb1956f1c79032a8"}
Nov 02 18:15:03.150498 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.150443    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" event={"ID":"beecd43d-55b7-4310-8d29-3d7db1d92847","Type":"ContainerStarted","Data":"db707eb03ff5e363290bd326e3ae1fda176c45d3d0bfeb90fb78a1ea86e73536"}
Nov 02 18:15:03.157396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.157362    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"]
Nov 02 18:15:03.160176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.160080    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerStarted","Data":"77f6aba08b48e78a298ad888e6397497fe0740e4710160250c2cb8b2fade9069"}
Nov 02 18:15:03.668109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:03.668030    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" podStartSLOduration=3.66800785 podStartE2EDuration="3.66800785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.199297776 +0000 UTC m=+2515.778707163" watchObservedRunningTime="2026-11-02 18:15:03.66800785 +0000 UTC m=+2516.247417237"
Nov 02 18:15:04.473565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.473493    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerStarted","Data":"c7c718a3f761d1fe956ddb73a4b13f70a25af4271a111582d30a1aad8c621cf6"}
Nov 02 18:15:04.514070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.513992    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" podStartSLOduration=4.513974386 podStartE2EDuration="4.513974386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:04.512266592 +0000 UTC m=+2517.091675978" watchObservedRunningTime="2026-11-02 18:15:04.513974386 +0000 UTC m=+2517.093383772"
Nov 02 18:15:04.531265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.530511    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerStarted","Data":"7c182f89e999191bc27078ab17f6b99f6327ec0576c1b1ffe33b3c60666efa22"}
Nov 02 18:15:04.532488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.531865    5042 scope.go:122] "RemoveContainer" containerID="7c182f89e999191bc27078ab17f6b99f6327ec0576c1b1ffe33b3c60666efa22"
Nov 02 18:15:04.553895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.553803    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerStarted","Data":"31f4beac70ccefe65272320769e7300cd201810c2d62d9fa05acd94952af0039"}
Nov 02 18:15:04.554376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.554234    5042 scope.go:122] "RemoveContainer" containerID="31f4beac70ccefe65272320769e7300cd201810c2d62d9fa05acd94952af0039"
Nov 02 18:15:04.601609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.601576    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" event={"ID":"7e6fff1c-409a-44c2-b849-63794da82212","Type":"ContainerStarted","Data":"2bb366d4247281b99798c86c157686904225c085f2f4e5444c3aefff1137a035"}
Nov 02 18:15:04.618360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.618276    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerStarted","Data":"9d17bb4cbb72f23e051975a1f0f1dfd8f6186242cb5f67301cbd2097f9a51486"}
Nov 02 18:15:04.640949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.640856    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" event={"ID":"40cccb0b-92ab-4399-a79d-65c5bc253aa0","Type":"ContainerStarted","Data":"784c488b64287e0799bc724784d42670b638557a6303b4e2aaf3a118ad0d1adc"}
Nov 02 18:15:04.645859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.644378    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" event={"ID":"beecd43d-55b7-4310-8d29-3d7db1d92847","Type":"ContainerStarted","Data":"5b0846f5e8c8aaab28c733633cc4db26b980d223b6fa6862fbac9f3ac93eadec"}
Nov 02 18:15:04.645859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:04.644918    5042 scope.go:122] "RemoveContainer" containerID="5b0846f5e8c8aaab28c733633cc4db26b980d223b6fa6862fbac9f3ac93eadec"
Nov 02 18:15:06.033902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.030679    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerStarted","Data":"3290d3cc324e10cbf2c53d47074ac6cf7864c9e1ee11f233d23ce22b9046a07e"}
Nov 02 18:15:06.099864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.090696    5042 generic.go:356] "Generic (PLEG): container finished" podID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerID="7c182f89e999191bc27078ab17f6b99f6327ec0576c1b1ffe33b3c60666efa22" exitCode=60
Nov 02 18:15:06.099864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.091136    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerDied","Data":"7c182f89e999191bc27078ab17f6b99f6327ec0576c1b1ffe33b3c60666efa22"}
Nov 02 18:15:06.099864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.091247    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerStarted","Data":"cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21"}
Nov 02 18:15:06.136706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.134082    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" podStartSLOduration=6.134056836 podStartE2EDuration="6.134056836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:06.090413848 +0000 UTC m=+2518.669823231" watchObservedRunningTime="2026-11-02 18:15:06.134056836 +0000 UTC m=+2518.713466222"
Nov 02 18:15:06.272061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.232014    5042 generic.go:356] "Generic (PLEG): container finished" podID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerID="31f4beac70ccefe65272320769e7300cd201810c2d62d9fa05acd94952af0039" exitCode=60
Nov 02 18:15:06.272061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.232112    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerDied","Data":"31f4beac70ccefe65272320769e7300cd201810c2d62d9fa05acd94952af0039"}
Nov 02 18:15:06.272061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.232165    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerStarted","Data":"c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"}
Nov 02 18:15:06.272061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.232193    5042 scope.go:122] "RemoveContainer" containerID="31f4beac70ccefe65272320769e7300cd201810c2d62d9fa05acd94952af0039"
Nov 02 18:15:06.272061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.232779    5042 scope.go:122] "RemoveContainer" containerID="c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"
Nov 02 18:15:06.272061 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:06.233345    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c_odh-ai-gateway-infra(2f9eff2e-1e2a-4472-b382-46a8fa8a995e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e"
Nov 02 18:15:06.727872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.725093    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerStarted","Data":"b9259041c1bd244d2cce6650d447ea8517f4ec451c9c3f1a620baab63881b49d"}
Nov 02 18:15:06.734798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.734743    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" podStartSLOduration=6.734726597 podStartE2EDuration="6.734726597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:06.205368151 +0000 UTC m=+2518.784777538" watchObservedRunningTime="2026-11-02 18:15:06.734726597 +0000 UTC m=+2519.314135984"
Nov 02 18:15:06.755118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.755070    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerStarted","Data":"a051ddcdb9a27c733197f6d051ccbf19624be1ba50328914dc5af33d46d7a02a"}
Nov 02 18:15:06.958333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.958294    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" event={"ID":"7e6fff1c-409a-44c2-b849-63794da82212","Type":"ContainerStarted","Data":"82b2c5d01f607cfeb5abb7fd4613a92daeadc373b11a62028a873fbd55b3992b"}
Nov 02 18:15:06.959113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.959087    5042 scope.go:122] "RemoveContainer" containerID="82b2c5d01f607cfeb5abb7fd4613a92daeadc373b11a62028a873fbd55b3992b"
Nov 02 18:15:06.993567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.993386    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerStarted","Data":"3c8a2d9cda722545b154870b0231097109a37479d0d9b57592a072b0ab733559"}
Nov 02 18:15:07.002180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:06.993859    5042 scope.go:122] "RemoveContainer" containerID="3c8a2d9cda722545b154870b0231097109a37479d0d9b57592a072b0ab733559"
Nov 02 18:15:07.362565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.206727    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:15:07.362565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.355222    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" podStartSLOduration=7.355198788 podStartE2EDuration="7.355198788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:06.764666074 +0000 UTC m=+2519.344075461" watchObservedRunningTime="2026-11-02 18:15:07.355198788 +0000 UTC m=+2519.934608176"
Nov 02 18:15:07.400238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.393738    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" podStartSLOduration=7.393717332 podStartE2EDuration="7.393717332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:06.808241121 +0000 UTC m=+2519.387650509" watchObservedRunningTime="2026-11-02 18:15:07.393717332 +0000 UTC m=+2519.973126717"
Nov 02 18:15:07.410998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.410965    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"]
Nov 02 18:15:07.411463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.411427    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy" containerID="cri-o://d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688" gracePeriod=30
Nov 02 18:15:07.433189 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.433148    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"]
Nov 02 18:15:07.481877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.469557    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13ff203f-nbc9t"]
Nov 02 18:15:07.673553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.671947    5042 scope.go:122] "RemoveContainer" containerID="b107d34f36ac5770d449cec5babe9780bde6767ccce27e3e45c1b9f67f8e30e6"
Nov 02 18:15:07.983363 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:07.983331    5042 scope.go:122] "RemoveContainer" containerID="5b10af9e69ab465c5ba6a8cb3e26b1f3cc5be0b36e7a4eb90fe29842cc92c0cd"
Nov 02 18:15:08.007772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.007722    5042 scope.go:122] "RemoveContainer" containerID="1937c8bf58ba48d41daa30cc0bc686f8b507da1a29e64a3c5c49bf02730aa9d9"
Nov 02 18:15:08.055097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.053232    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39fe532f-9e72-4297-9278-d261e3e24111" path="/var/lib/kubelet/pods/39fe532f-9e72-4297-9278-d261e3e24111/volumes"
Nov 02 18:15:08.055097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.053920    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" event={"ID":"40cccb0b-92ab-4399-a79d-65c5bc253aa0","Type":"ContainerStarted","Data":"b107d34f36ac5770d449cec5babe9780bde6767ccce27e3e45c1b9f67f8e30e6"}
Nov 02 18:15:08.055097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.053995    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" event={"ID":"8431f13a-72b7-418f-8563-10812bf2c419","Type":"ContainerStarted","Data":"5b10af9e69ab465c5ba6a8cb3e26b1f3cc5be0b36e7a4eb90fe29842cc92c0cd"}
Nov 02 18:15:08.055097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.054038    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" event={"ID":"ce6bc700-de9b-4e95-8d3a-a704283cd4fc","Type":"ContainerStarted","Data":"1937c8bf58ba48d41daa30cc0bc686f8b507da1a29e64a3c5c49bf02730aa9d9"}
Nov 02 18:15:08.065884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.064034    5042 generic.go:356] "Generic (PLEG): container finished" podID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerID="5b0846f5e8c8aaab28c733633cc4db26b980d223b6fa6862fbac9f3ac93eadec" exitCode=60
Nov 02 18:15:08.065884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.065647    5042 scope.go:122] "RemoveContainer" containerID="2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037"
Nov 02 18:15:08.067932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.067892    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" event={"ID":"beecd43d-55b7-4310-8d29-3d7db1d92847","Type":"ContainerDied","Data":"5b0846f5e8c8aaab28c733633cc4db26b980d223b6fa6862fbac9f3ac93eadec"}
Nov 02 18:15:08.068145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.068127    5042 scope.go:122] "RemoveContainer" containerID="5b0846f5e8c8aaab28c733633cc4db26b980d223b6fa6862fbac9f3ac93eadec"
Nov 02 18:15:08.089984 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:08.077099    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54_odh-ai-gateway-infra(beecd43d-55b7-4310-8d29-3d7db1d92847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847"
Nov 02 18:15:08.117303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.117120    5042 generic.go:356] "Generic (PLEG): container finished" podID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerID="c7c718a3f761d1fe956ddb73a4b13f70a25af4271a111582d30a1aad8c621cf6" exitCode=60
Nov 02 18:15:08.117303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.117255    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerDied","Data":"c7c718a3f761d1fe956ddb73a4b13f70a25af4271a111582d30a1aad8c621cf6"}
Nov 02 18:15:08.121751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.121164    5042 scope.go:122] "RemoveContainer" containerID="c7c718a3f761d1fe956ddb73a4b13f70a25af4271a111582d30a1aad8c621cf6"
Nov 02 18:15:08.530142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.530114    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 18:15:08.794673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.794631    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.794943    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.794988    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.795029    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.795172    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.795270    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.795325    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.795385    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:08.795799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.795542    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:09.018298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.950057    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:09.018298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:08.953965    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.018298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.011266    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.018298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.018170    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.047543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.041086    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.125203    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.125256    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.125295    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\") pod \"4d370441-564c-40be-8e46-a89c086bdd05\" (UID: \"4d370441-564c-40be-8e46-a89c086bdd05\") "
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.126430    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.126458    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:09.126557    5042 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4d370441-564c-40be-8e46-a89c086bdd05/volumes/kubernetes.io~empty-dir/workload-socket
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.126575    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:09.126628    5042 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4d370441-564c-40be-8e46-a89c086bdd05/volumes/kubernetes.io~empty-dir/credential-socket
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.126638    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:09.126699    5042 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4d370441-564c-40be-8e46-a89c086bdd05/volumes/kubernetes.io~configmap/istiod-ca-cert
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.126709    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.155604    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:15:09.175482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.172012    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "kube-api-access-9qlx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:09.217676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.191015    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:09.248396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.227321    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy" pod "4d370441-564c-40be-8e46-a89c086bdd05" (UID: "4d370441-564c-40be-8e46-a89c086bdd05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.248396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.228451    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d370441-564c-40be-8e46-a89c086bdd05-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.248396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.247623    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.248396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.247645    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d370441-564c-40be-8e46-a89c086bdd05-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.248396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.247661    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.248396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.247676    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qlx6\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-kube-api-access-9qlx6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.262195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.253470    5042 generic.go:356] "Generic (PLEG): container finished" podID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerID="b107d34f36ac5770d449cec5babe9780bde6767ccce27e3e45c1b9f67f8e30e6" exitCode=60
Nov 02 18:15:09.262195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.253579    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" event={"ID":"40cccb0b-92ab-4399-a79d-65c5bc253aa0","Type":"ContainerDied","Data":"b107d34f36ac5770d449cec5babe9780bde6767ccce27e3e45c1b9f67f8e30e6"}
Nov 02 18:15:09.263645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.262403    5042 generic.go:356] "Generic (PLEG): container finished" podID="8431f13a-72b7-418f-8563-10812bf2c419" containerID="5b10af9e69ab465c5ba6a8cb3e26b1f3cc5be0b36e7a4eb90fe29842cc92c0cd" exitCode=60
Nov 02 18:15:09.263645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.262498    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" event={"ID":"8431f13a-72b7-418f-8563-10812bf2c419","Type":"ContainerDied","Data":"5b10af9e69ab465c5ba6a8cb3e26b1f3cc5be0b36e7a4eb90fe29842cc92c0cd"}
Nov 02 18:15:09.288540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.288455    5042 generic.go:356] "Generic (PLEG): container finished" podID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerID="1937c8bf58ba48d41daa30cc0bc686f8b507da1a29e64a3c5c49bf02730aa9d9" exitCode=60
Nov 02 18:15:09.288962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.288680    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" event={"ID":"ce6bc700-de9b-4e95-8d3a-a704283cd4fc","Type":"ContainerDied","Data":"1937c8bf58ba48d41daa30cc0bc686f8b507da1a29e64a3c5c49bf02730aa9d9"}
Nov 02 18:15:09.351157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.351123    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d370441-564c-40be-8e46-a89c086bdd05-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.351410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.351391    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d370441-564c-40be-8e46-a89c086bdd05-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:09.374913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.374768    5042 generic.go:356] "Generic (PLEG): container finished" podID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerID="2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037" exitCode=60
Nov 02 18:15:09.376085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.375372    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" event={"ID":"beecd43d-55b7-4310-8d29-3d7db1d92847","Type":"ContainerDied","Data":"2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037"}
Nov 02 18:15:09.379952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.376545    5042 scope.go:122] "RemoveContainer" containerID="2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037"
Nov 02 18:15:09.379952 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:09.377046    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54_odh-ai-gateway-infra(beecd43d-55b7-4310-8d29-3d7db1d92847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847"
Nov 02 18:15:09.388072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.388025    5042 generic.go:356] "Generic (PLEG): container finished" podID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerID="3290d3cc324e10cbf2c53d47074ac6cf7864c9e1ee11f233d23ce22b9046a07e" exitCode=60
Nov 02 18:15:09.388513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.388491    5042 scope.go:122] "RemoveContainer" containerID="3290d3cc324e10cbf2c53d47074ac6cf7864c9e1ee11f233d23ce22b9046a07e"
Nov 02 18:15:09.390548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.388770    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerDied","Data":"3290d3cc324e10cbf2c53d47074ac6cf7864c9e1ee11f233d23ce22b9046a07e"}
Nov 02 18:15:09.395168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.393957    5042 generic.go:356] "Generic (PLEG): container finished" podID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerID="cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21" exitCode=60
Nov 02 18:15:09.395168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.394100    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerDied","Data":"cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21"}
Nov 02 18:15:09.395168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.394153    5042 scope.go:122] "RemoveContainer" containerID="7c182f89e999191bc27078ab17f6b99f6327ec0576c1b1ffe33b3c60666efa22"
Nov 02 18:15:09.395168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.394347    5042 scope.go:122] "RemoveContainer" containerID="cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21"
Nov 02 18:15:09.395168 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:09.394649    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk_odh-ai-gateway-infra(0d7b295a-1204-4db7-81dd-bb7483eff02d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d"
Nov 02 18:15:09.397999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.397981    5042 generic.go:356] "Generic (PLEG): container finished" podID="4d370441-564c-40be-8e46-a89c086bdd05" containerID="9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74" exitCode=0
Nov 02 18:15:09.398169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.398158    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"
Nov 02 18:15:09.398260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.398240    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" event={"ID":"4d370441-564c-40be-8e46-a89c086bdd05","Type":"ContainerDied","Data":"9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74"}
Nov 02 18:15:09.398301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.398282    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z" event={"ID":"4d370441-564c-40be-8e46-a89c086bdd05","Type":"ContainerDied","Data":"c7c7302e6cb32d74b1a44b4d46173f00701593f7871eeaafe13441e3b5f866c2"}
Nov 02 18:15:09.416597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.416542    5042 generic.go:356] "Generic (PLEG): container finished" podID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerID="c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c" exitCode=60
Nov 02 18:15:09.416735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.416654    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerDied","Data":"c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"}
Nov 02 18:15:09.417344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.416996    5042 scope.go:122] "RemoveContainer" containerID="c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"
Nov 02 18:15:09.417483 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:09.417351    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c_odh-ai-gateway-infra(2f9eff2e-1e2a-4472-b382-46a8fa8a995e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e"
Nov 02 18:15:09.419312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.419296    5042 generic.go:356] "Generic (PLEG): container finished" podID="3baacd62-a713-4016-850a-87cd47a0aa89" containerID="b9259041c1bd244d2cce6650d447ea8517f4ec451c9c3f1a620baab63881b49d" exitCode=60
Nov 02 18:15:09.419421 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.419342    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerDied","Data":"b9259041c1bd244d2cce6650d447ea8517f4ec451c9c3f1a620baab63881b49d"}
Nov 02 18:15:09.420051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.419683    5042 scope.go:122] "RemoveContainer" containerID="b9259041c1bd244d2cce6650d447ea8517f4ec451c9c3f1a620baab63881b49d"
Nov 02 18:15:09.430764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.424344    5042 generic.go:356] "Generic (PLEG): container finished" podID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerID="a051ddcdb9a27c733197f6d051ccbf19624be1ba50328914dc5af33d46d7a02a" exitCode=60
Nov 02 18:15:09.430764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.425470    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerDied","Data":"a051ddcdb9a27c733197f6d051ccbf19624be1ba50328914dc5af33d46d7a02a"}
Nov 02 18:15:09.430764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.425861    5042 scope.go:122] "RemoveContainer" containerID="a051ddcdb9a27c733197f6d051ccbf19624be1ba50328914dc5af33d46d7a02a"
Nov 02 18:15:09.434874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.434107    5042 generic.go:356] "Generic (PLEG): container finished" podID="7e6fff1c-409a-44c2-b849-63794da82212" containerID="82b2c5d01f607cfeb5abb7fd4613a92daeadc373b11a62028a873fbd55b3992b" exitCode=60
Nov 02 18:15:09.434874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.434175    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" event={"ID":"7e6fff1c-409a-44c2-b849-63794da82212","Type":"ContainerDied","Data":"82b2c5d01f607cfeb5abb7fd4613a92daeadc373b11a62028a873fbd55b3992b"}
Nov 02 18:15:09.434874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.434482    5042 scope.go:122] "RemoveContainer" containerID="3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52"
Nov 02 18:15:09.434874 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:09.434745    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp_odh-ai-gateway-infra(7e6fff1c-409a-44c2-b849-63794da82212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" podUID="7e6fff1c-409a-44c2-b849-63794da82212"
Nov 02 18:15:09.437069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.437048    5042 generic.go:356] "Generic (PLEG): container finished" podID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerID="3c8a2d9cda722545b154870b0231097109a37479d0d9b57592a072b0ab733559" exitCode=60
Nov 02 18:15:09.438006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.437235    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerDied","Data":"3c8a2d9cda722545b154870b0231097109a37479d0d9b57592a072b0ab733559"}
Nov 02 18:15:09.445126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.445101    5042 scope.go:122] "RemoveContainer" containerID="9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74"
Nov 02 18:15:09.499455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.499413    5042 scope.go:122] "RemoveContainer" containerID="9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74"
Nov 02 18:15:09.505184 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:09.505146    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74\": container with ID starting with 9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74 not found: ID does not exist" containerID="9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74"
Nov 02 18:15:09.505288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.505199    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74"} err="failed to get container status \"9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74\": rpc error: code = NotFound desc = could not find container \"9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74\": container with ID starting with 9e4bd946d6a35ec5dd3bcb36856b7183429446d0fec87f100bc23cac3fe0ec74 not found: ID does not exist"
Nov 02 18:15:09.505288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.505229    5042 scope.go:122] "RemoveContainer" containerID="82b2c5d01f607cfeb5abb7fd4613a92daeadc373b11a62028a873fbd55b3992b"
Nov 02 18:15:09.552807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.552572    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" podStartSLOduration=9.552551905 podStartE2EDuration="9.552551905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:09.545948647 +0000 UTC m=+2522.125358034" watchObservedRunningTime="2026-11-02 18:15:09.552551905 +0000 UTC m=+2522.131961294"
Nov 02 18:15:09.570359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.570330    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"]
Nov 02 18:15:09.581014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:09.580980    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-fmn8z"]
Nov 02 18:15:10.457450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.457396    5042 generic.go:356] "Generic (PLEG): container finished" podID="7e6fff1c-409a-44c2-b849-63794da82212" containerID="3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52" exitCode=60
Nov 02 18:15:10.458170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.457539    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" event={"ID":"7e6fff1c-409a-44c2-b849-63794da82212","Type":"ContainerDied","Data":"3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52"}
Nov 02 18:15:10.459112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.458179    5042 scope.go:122] "RemoveContainer" containerID="3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52"
Nov 02 18:15:10.459112 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.458858    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp_odh-ai-gateway-infra(7e6fff1c-409a-44c2-b849-63794da82212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" podUID="7e6fff1c-409a-44c2-b849-63794da82212"
Nov 02 18:15:10.461109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.461081    5042 generic.go:356] "Generic (PLEG): container finished" podID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerID="646da653fbafb373b7fe60e17ca02ad58ff26f317d5df3d3b3a04fb7ae0ff809" exitCode=60
Nov 02 18:15:10.461302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.461196    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerDied","Data":"646da653fbafb373b7fe60e17ca02ad58ff26f317d5df3d3b3a04fb7ae0ff809"}
Nov 02 18:15:10.461302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.461254    5042 scope.go:122] "RemoveContainer" containerID="3c8a2d9cda722545b154870b0231097109a37479d0d9b57592a072b0ab733559"
Nov 02 18:15:10.462474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.462454    5042 scope.go:122] "RemoveContainer" containerID="646da653fbafb373b7fe60e17ca02ad58ff26f317d5df3d3b3a04fb7ae0ff809"
Nov 02 18:15:10.462933 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.462900    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2_odh-ai-gateway-infra(b940ad60-3ce2-4bc7-8fa3-2346718ee228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228"
Nov 02 18:15:10.465632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.465497    5042 generic.go:356] "Generic (PLEG): container finished" podID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerID="c559d5926cc408961cf933c7b83c2cc0a523eae6a829cb84cd4503df3e430f9f" exitCode=60
Nov 02 18:15:10.465632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.465537    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" event={"ID":"40cccb0b-92ab-4399-a79d-65c5bc253aa0","Type":"ContainerDied","Data":"c559d5926cc408961cf933c7b83c2cc0a523eae6a829cb84cd4503df3e430f9f"}
Nov 02 18:15:10.466163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.466126    5042 scope.go:122] "RemoveContainer" containerID="c559d5926cc408961cf933c7b83c2cc0a523eae6a829cb84cd4503df3e430f9f"
Nov 02 18:15:10.466927 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.466897    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v_odh-ai-gateway-infra(40cccb0b-92ab-4399-a79d-65c5bc253aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0"
Nov 02 18:15:10.468601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.468424    5042 generic.go:356] "Generic (PLEG): container finished" podID="8431f13a-72b7-418f-8563-10812bf2c419" containerID="1f6ebb0c67e885e6c640a54389511b3d369d54c45dfdc28cf0f68f2827fca00f" exitCode=60
Nov 02 18:15:10.468601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.468508    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" event={"ID":"8431f13a-72b7-418f-8563-10812bf2c419","Type":"ContainerDied","Data":"1f6ebb0c67e885e6c640a54389511b3d369d54c45dfdc28cf0f68f2827fca00f"}
Nov 02 18:15:10.468913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.468825    5042 scope.go:122] "RemoveContainer" containerID="1f6ebb0c67e885e6c640a54389511b3d369d54c45dfdc28cf0f68f2827fca00f"
Nov 02 18:15:10.469354 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.469256    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx_odh-ai-gateway-infra(8431f13a-72b7-418f-8563-10812bf2c419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" podUID="8431f13a-72b7-418f-8563-10812bf2c419"
Nov 02 18:15:10.471606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.471586    5042 generic.go:356] "Generic (PLEG): container finished" podID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerID="f9de8a602c8c65ee56ddf1f798acd547eccd987f92862c0f5e042698b0b98427" exitCode=60
Nov 02 18:15:10.471802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.471771    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" event={"ID":"ce6bc700-de9b-4e95-8d3a-a704283cd4fc","Type":"ContainerDied","Data":"f9de8a602c8c65ee56ddf1f798acd547eccd987f92862c0f5e042698b0b98427"}
Nov 02 18:15:10.477890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.472077    5042 scope.go:122] "RemoveContainer" containerID="f9de8a602c8c65ee56ddf1f798acd547eccd987f92862c0f5e042698b0b98427"
Nov 02 18:15:10.477890 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.472497    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6_odh-ai-gateway-infra(ce6bc700-de9b-4e95-8d3a-a704283cd4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc"
Nov 02 18:15:10.512221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.512183    5042 scope.go:122] "RemoveContainer" containerID="b107d34f36ac5770d449cec5babe9780bde6767ccce27e3e45c1b9f67f8e30e6"
Nov 02 18:15:10.516589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.516527    5042 generic.go:356] "Generic (PLEG): container finished" podID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerID="254f7c3011273915825c3c8bffd3a9b972128cbfa4fb4848eba382a673deeb98" exitCode=60
Nov 02 18:15:10.516905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.516822    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerDied","Data":"254f7c3011273915825c3c8bffd3a9b972128cbfa4fb4848eba382a673deeb98"}
Nov 02 18:15:10.517379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.517350    5042 scope.go:122] "RemoveContainer" containerID="254f7c3011273915825c3c8bffd3a9b972128cbfa4fb4848eba382a673deeb98"
Nov 02 18:15:10.517787 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.517754    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4_odh-ai-gateway-infra(92658c40-3651-4c67-a106-b0ed6b13fb9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e"
Nov 02 18:15:10.522568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.522542    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerStarted","Data":"a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd"}
Nov 02 18:15:10.529498 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.529478    5042 scope.go:122] "RemoveContainer" containerID="cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21"
Nov 02 18:15:10.534512 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.534482    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk_odh-ai-gateway-infra(0d7b295a-1204-4db7-81dd-bb7483eff02d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d"
Nov 02 18:15:10.540173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.540145    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerStarted","Data":"6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac"}
Nov 02 18:15:10.554663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.554631    5042 scope.go:122] "RemoveContainer" containerID="5b10af9e69ab465c5ba6a8cb3e26b1f3cc5be0b36e7a4eb90fe29842cc92c0cd"
Nov 02 18:15:10.568368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.567951    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerStarted","Data":"415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6"}
Nov 02 18:15:10.570137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.570088    5042 scope.go:122] "RemoveContainer" containerID="c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"
Nov 02 18:15:10.572590 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.570494    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c_odh-ai-gateway-infra(2f9eff2e-1e2a-4472-b382-46a8fa8a995e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e"
Nov 02 18:15:10.573951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.573518    5042 scope.go:122] "RemoveContainer" containerID="2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037"
Nov 02 18:15:10.574170 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:10.574144    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54_odh-ai-gateway-infra(beecd43d-55b7-4310-8d29-3d7db1d92847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847"
Nov 02 18:15:10.619337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.608936    5042 scope.go:122] "RemoveContainer" containerID="1937c8bf58ba48d41daa30cc0bc686f8b507da1a29e64a3c5c49bf02730aa9d9"
Nov 02 18:15:10.671530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:10.671308    5042 scope.go:122] "RemoveContainer" containerID="c7c718a3f761d1fe956ddb73a4b13f70a25af4271a111582d30a1aad8c621cf6"
Nov 02 18:15:11.547534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.547504    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d370441-564c-40be-8e46-a89c086bdd05" path="/var/lib/kubelet/pods/4d370441-564c-40be-8e46-a89c086bdd05/volumes"
Nov 02 18:15:11.584690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.584652    5042 generic.go:356] "Generic (PLEG): container finished" podID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerID="a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd" exitCode=60
Nov 02 18:15:11.585231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.584790    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerDied","Data":"a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd"}
Nov 02 18:15:11.585231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.584887    5042 scope.go:122] "RemoveContainer" containerID="3290d3cc324e10cbf2c53d47074ac6cf7864c9e1ee11f233d23ce22b9046a07e"
Nov 02 18:15:11.586116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.585394    5042 scope.go:122] "RemoveContainer" containerID="a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd"
Nov 02 18:15:11.586116 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.585889    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm_odh-ai-gateway-infra(f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf"
Nov 02 18:15:11.591032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.591002    5042 generic.go:356] "Generic (PLEG): container finished" podID="3baacd62-a713-4016-850a-87cd47a0aa89" containerID="6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac" exitCode=60
Nov 02 18:15:11.591250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.591088    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerDied","Data":"6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac"}
Nov 02 18:15:11.591621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.591597    5042 scope.go:122] "RemoveContainer" containerID="6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac"
Nov 02 18:15:11.592337 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.592298    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-sjzz6_odh-ai-gateway-infra(3baacd62-a713-4016-850a-87cd47a0aa89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" podUID="3baacd62-a713-4016-850a-87cd47a0aa89"
Nov 02 18:15:11.596479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.596451    5042 generic.go:356] "Generic (PLEG): container finished" podID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerID="415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6" exitCode=60
Nov 02 18:15:11.597087 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.596912    5042 scope.go:122] "RemoveContainer" containerID="415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6"
Nov 02 18:15:11.597316 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.597291    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg_odh-ai-gateway-infra(f97c3798-d7b7-4b99-9d2c-3364239923aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa"
Nov 02 18:15:11.598437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.597344    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerDied","Data":"415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6"}
Nov 02 18:15:11.601311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.601287    5042 scope.go:122] "RemoveContainer" containerID="646da653fbafb373b7fe60e17ca02ad58ff26f317d5df3d3b3a04fb7ae0ff809"
Nov 02 18:15:11.601883 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.601704    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2_odh-ai-gateway-infra(b940ad60-3ce2-4bc7-8fa3-2346718ee228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228"
Nov 02 18:15:11.604611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.604587    5042 scope.go:122] "RemoveContainer" containerID="c559d5926cc408961cf933c7b83c2cc0a523eae6a829cb84cd4503df3e430f9f"
Nov 02 18:15:11.604949 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.604924    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v_odh-ai-gateway-infra(40cccb0b-92ab-4399-a79d-65c5bc253aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0"
Nov 02 18:15:11.613057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.613030    5042 scope.go:122] "RemoveContainer" containerID="b9259041c1bd244d2cce6650d447ea8517f4ec451c9c3f1a620baab63881b49d"
Nov 02 18:15:11.614639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.614613    5042 scope.go:122] "RemoveContainer" containerID="1f6ebb0c67e885e6c640a54389511b3d369d54c45dfdc28cf0f68f2827fca00f"
Nov 02 18:15:11.615175 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.615129    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx_odh-ai-gateway-infra(8431f13a-72b7-418f-8563-10812bf2c419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" podUID="8431f13a-72b7-418f-8563-10812bf2c419"
Nov 02 18:15:11.616682 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.616661    5042 scope.go:122] "RemoveContainer" containerID="f9de8a602c8c65ee56ddf1f798acd547eccd987f92862c0f5e042698b0b98427"
Nov 02 18:15:11.617052 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.617026    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6_odh-ai-gateway-infra(ce6bc700-de9b-4e95-8d3a-a704283cd4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc"
Nov 02 18:15:11.619045 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.618880    5042 scope.go:122] "RemoveContainer" containerID="254f7c3011273915825c3c8bffd3a9b972128cbfa4fb4848eba382a673deeb98"
Nov 02 18:15:11.619218 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.619193    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4_odh-ai-gateway-infra(92658c40-3651-4c67-a106-b0ed6b13fb9e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e"
Nov 02 18:15:11.619394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.619370    5042 scope.go:122] "RemoveContainer" containerID="3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52"
Nov 02 18:15:11.619700 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:11.619679    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp_odh-ai-gateway-infra(7e6fff1c-409a-44c2-b849-63794da82212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" podUID="7e6fff1c-409a-44c2-b849-63794da82212"
Nov 02 18:15:11.635216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:11.635192    5042 scope.go:122] "RemoveContainer" containerID="a051ddcdb9a27c733197f6d051ccbf19624be1ba50328914dc5af33d46d7a02a"
Nov 02 18:15:12.633088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:12.633046    5042 scope.go:122] "RemoveContainer" containerID="a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd"
Nov 02 18:15:12.634136 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:12.633556    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm_odh-ai-gateway-infra(f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf"
Nov 02 18:15:12.634988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:12.634964    5042 scope.go:122] "RemoveContainer" containerID="6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac"
Nov 02 18:15:12.635366 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:12.635336    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-sjzz6_odh-ai-gateway-infra(3baacd62-a713-4016-850a-87cd47a0aa89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" podUID="3baacd62-a713-4016-850a-87cd47a0aa89"
Nov 02 18:15:12.636984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:12.636965    5042 scope.go:122] "RemoveContainer" containerID="415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6"
Nov 02 18:15:12.637299 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:12.637280    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg_odh-ai-gateway-infra(f97c3798-d7b7-4b99-9d2c-3364239923aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa"
Nov 02 18:15:13.224250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.224217    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:15:13.337927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.337800    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.337927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.337918    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.337967    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.338043    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.338074    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.338116    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.338188    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.338301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.338496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.338398    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket\") pod \"5f1ea13b-c308-408a-83fd-bc29c2240d18\" (UID: \"5f1ea13b-c308-408a-83fd-bc29c2240d18\") "
Nov 02 18:15:13.339175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.339048    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:13.339408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.339383    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1ea13b-c308-408a-83fd-bc29c2240d18-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.339802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.339779    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:13.340017 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.339860    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:13.340107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.340004    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:13.340256 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.340236    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:13.341350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.341308    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:15:13.342081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.342056    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:13.342419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.342402    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:13.345587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.345550    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb" pod "5f1ea13b-c308-408a-83fd-bc29c2240d18" (UID: "5f1ea13b-c308-408a-83fd-bc29c2240d18"). InnerVolumeSpecName "kube-api-access-jqjlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:13.441360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441325    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441599    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjlb\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-kube-api-access-jqjlb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441627    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441639    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441651    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441676    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1ea13b-c308-408a-83fd-bc29c2240d18-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441690    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.441783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.441702    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1ea13b-c308-408a-83fd-bc29c2240d18-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:13.649133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.649035    5042 generic.go:356] "Generic (PLEG): container finished" podID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerID="d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688" exitCode=0
Nov 02 18:15:13.650199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.649160    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"
Nov 02 18:15:13.650199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.649180    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" event={"ID":"5f1ea13b-c308-408a-83fd-bc29c2240d18","Type":"ContainerDied","Data":"d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688"}
Nov 02 18:15:13.650199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.649252    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f" event={"ID":"5f1ea13b-c308-408a-83fd-bc29c2240d18","Type":"ContainerDied","Data":"1d6ee9e45e83e69dcc704c9ccff08cdb3dee2fceb1fde8dc6078988e11a9a9df"}
Nov 02 18:15:13.650199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.649290    5042 scope.go:122] "RemoveContainer" containerID="d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688"
Nov 02 18:15:13.665977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.665953    5042 scope.go:122] "RemoveContainer" containerID="d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688"
Nov 02 18:15:13.666380 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:13.666351    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688\": container with ID starting with d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688 not found: ID does not exist" containerID="d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688"
Nov 02 18:15:13.667289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.666389    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688"} err="failed to get container status \"d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688\": rpc error: code = NotFound desc = could not find container \"d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688\": container with ID starting with d0d821d03f09634888ec85d761aa9366338c4ba64377fd54e99a8de5ac62e688 not found: ID does not exist"
Nov 02 18:15:13.683862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.682605    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"]
Nov 02 18:15:13.691175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:13.691149    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-jf85f"]
Nov 02 18:15:15.549237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.549201    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" path="/var/lib/kubelet/pods/5f1ea13b-c308-408a-83fd-bc29c2240d18/volumes"
Nov 02 18:15:15.642907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.642869    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"]
Nov 02 18:15:15.643908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.643883    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy"
Nov 02 18:15:15.643908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.643905    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy"
Nov 02 18:15:15.644150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.643923    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy"
Nov 02 18:15:15.644150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.643928    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy"
Nov 02 18:15:15.644368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.644347    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d370441-564c-40be-8e46-a89c086bdd05" containerName="istio-proxy"
Nov 02 18:15:15.644446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.644373    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1ea13b-c308-408a-83fd-bc29c2240d18" containerName="istio-proxy"
Nov 02 18:15:15.650129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.650100    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:15.668741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.667466    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"]
Nov 02 18:15:15.700145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.700111    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-312f21-dockercfg-q7fmb\""
Nov 02 18:15:15.700350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.700202    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-312f21\""
Nov 02 18:15:15.735073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.734051    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:15.735073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.734329    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nkzz\" (UniqueName: \"kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz\") pod \"payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:15.862917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.862309    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nkzz\" (UniqueName: \"kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz\") pod \"payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:15.862917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.862533    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:15.892871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:15.863539    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:16.217872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:16.217797    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nkzz\" (UniqueName: \"kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz\") pod \"payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:16.289332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:16.289294    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:16.536692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:16.536612    5042 scope.go:122] "RemoveContainer" containerID="cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21"
Nov 02 18:15:16.597779 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:16.597743    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd745b23_0598_4e19_9537_3ed5c422891e.slice/crio-95534bfacda8fd77c81c53fbc869d855050afbd1d69b40a8a6b171a1882b8d19 WatchSource:0}: Error finding container 95534bfacda8fd77c81c53fbc869d855050afbd1d69b40a8a6b171a1882b8d19: Status 404 returned error can't find the container with id 95534bfacda8fd77c81c53fbc869d855050afbd1d69b40a8a6b171a1882b8d19
Nov 02 18:15:16.599935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:16.599907    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"]
Nov 02 18:15:16.749022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:16.748389    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" event={"ID":"dd745b23-0598-4e19-9537-3ed5c422891e","Type":"ContainerStarted","Data":"95534bfacda8fd77c81c53fbc869d855050afbd1d69b40a8a6b171a1882b8d19"}
Nov 02 18:15:17.760033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:17.759997    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" event={"ID":"dd745b23-0598-4e19-9537-3ed5c422891e","Type":"ContainerStarted","Data":"6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd"}
Nov 02 18:15:17.761196 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:17.760094    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:17.762431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:17.762409    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerStarted","Data":"b475995ac69f78b68d514c2ccf89ae4e4d3ea0a44a15e3f50048ca0d8bae43dd"}
Nov 02 18:15:17.785984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:17.785929    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" podStartSLOduration=2.785909209 podStartE2EDuration="2.785909209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:17.776441783 +0000 UTC m=+2530.355851167" watchObservedRunningTime="2026-11-02 18:15:17.785909209 +0000 UTC m=+2530.365318596"
Nov 02 18:15:18.045090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.044995    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"]
Nov 02 18:15:18.049326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.049292    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc384ee-644z9"]
Nov 02 18:15:18.536172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.536136    5042 scope.go:122] "RemoveContainer" containerID="c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"
Nov 02 18:15:18.580945 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:18.580902    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:15:18.776114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.776079    5042 generic.go:356] "Generic (PLEG): container finished" podID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerID="b475995ac69f78b68d514c2ccf89ae4e4d3ea0a44a15e3f50048ca0d8bae43dd" exitCode=60
Nov 02 18:15:18.776630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.776167    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerDied","Data":"b475995ac69f78b68d514c2ccf89ae4e4d3ea0a44a15e3f50048ca0d8bae43dd"}
Nov 02 18:15:18.776630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.776233    5042 scope.go:122] "RemoveContainer" containerID="cf52d3bdbd7ac6a8f4a0181ad4022977cd9d559508e1176fa5ca2b4a7095ff21"
Nov 02 18:15:18.777271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.777141    5042 scope.go:122] "RemoveContainer" containerID="b475995ac69f78b68d514c2ccf89ae4e4d3ea0a44a15e3f50048ca0d8bae43dd"
Nov 02 18:15:18.777553 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:18.777527    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk_odh-ai-gateway-infra(0d7b295a-1204-4db7-81dd-bb7483eff02d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d"
Nov 02 18:15:18.805776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:18.805749    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"]
Nov 02 18:15:19.536305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:19.536266    5042 scope.go:122] "RemoveContainer" containerID="2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037"
Nov 02 18:15:19.553146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:19.553113    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b33c59-006a-4c3f-953a-12bc953149e1" path="/var/lib/kubelet/pods/19b33c59-006a-4c3f-953a-12bc953149e1/volumes"
Nov 02 18:15:19.795556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:19.795521    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerStarted","Data":"be2b613ff5c473ffbb676ed819892c7de756be116d484f7e2331e09d1d5f4a07"}
Nov 02 18:15:19.796177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:19.796141    5042 scope.go:122] "RemoveContainer" containerID="be2b613ff5c473ffbb676ed819892c7de756be116d484f7e2331e09d1d5f4a07"
Nov 02 18:15:19.796656 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:19.796632    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c_odh-ai-gateway-infra(2f9eff2e-1e2a-4472-b382-46a8fa8a995e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e"
Nov 02 18:15:20.137084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.137057    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:20.222066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.222023    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7\" (UniqueName: \"kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7\") pod \"0d7b295a-1204-4db7-81dd-bb7483eff02d\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") "
Nov 02 18:15:20.222284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.222146    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca\") pod \"0d7b295a-1204-4db7-81dd-bb7483eff02d\" (UID: \"0d7b295a-1204-4db7-81dd-bb7483eff02d\") "
Nov 02 18:15:20.222586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.222550    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca" pod "0d7b295a-1204-4db7-81dd-bb7483eff02d" (UID: "0d7b295a-1204-4db7-81dd-bb7483eff02d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:20.223341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.223316    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d7b295a-1204-4db7-81dd-bb7483eff02d-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:20.225304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.225276    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7" pod "0d7b295a-1204-4db7-81dd-bb7483eff02d" (UID: "0d7b295a-1204-4db7-81dd-bb7483eff02d"). InnerVolumeSpecName "kube-api-access-w29c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:20.325775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.325730    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w29c7\" (UniqueName: \"kubernetes.io/projected/0d7b295a-1204-4db7-81dd-bb7483eff02d-kube-api-access-w29c7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:20.537176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.537144    5042 scope.go:122] "RemoveContainer" containerID="f9de8a602c8c65ee56ddf1f798acd547eccd987f92862c0f5e042698b0b98427"
Nov 02 18:15:20.537546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.537520    5042 scope.go:122] "RemoveContainer" containerID="c559d5926cc408961cf933c7b83c2cc0a523eae6a829cb84cd4503df3e430f9f"
Nov 02 18:15:20.919862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.911267    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk" event={"ID":"0d7b295a-1204-4db7-81dd-bb7483eff02d","Type":"ContainerDied","Data":"0019f8588eb7be7a82fba4063f30ae636b409708278b63907c35181369e0b700"}
Nov 02 18:15:20.919862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.911348    5042 scope.go:122] "RemoveContainer" containerID="b475995ac69f78b68d514c2ccf89ae4e4d3ea0a44a15e3f50048ca0d8bae43dd"
Nov 02 18:15:20.919862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.911519    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"
Nov 02 18:15:20.937600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.935584    5042 generic.go:356] "Generic (PLEG): container finished" podID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerID="be2b613ff5c473ffbb676ed819892c7de756be116d484f7e2331e09d1d5f4a07" exitCode=60
Nov 02 18:15:20.937600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.935805    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerDied","Data":"be2b613ff5c473ffbb676ed819892c7de756be116d484f7e2331e09d1d5f4a07"}
Nov 02 18:15:20.937600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.936371    5042 scope.go:122] "RemoveContainer" containerID="be2b613ff5c473ffbb676ed819892c7de756be116d484f7e2331e09d1d5f4a07"
Nov 02 18:15:20.937600 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:20.936952    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c_odh-ai-gateway-infra(2f9eff2e-1e2a-4472-b382-46a8fa8a995e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e"
Nov 02 18:15:20.942042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.942021    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"]
Nov 02 18:15:20.943258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.943237    5042 generic.go:356] "Generic (PLEG): container finished" podID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerID="d6379936c1ad1655384f8afa368dd5f9377e001ad93154dfc7fdaa3afde13380" exitCode=60
Nov 02 18:15:20.943571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.943553    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" event={"ID":"beecd43d-55b7-4310-8d29-3d7db1d92847","Type":"ContainerDied","Data":"d6379936c1ad1655384f8afa368dd5f9377e001ad93154dfc7fdaa3afde13380"}
Nov 02 18:15:20.944224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.944206    5042 scope.go:122] "RemoveContainer" containerID="d6379936c1ad1655384f8afa368dd5f9377e001ad93154dfc7fdaa3afde13380"
Nov 02 18:15:20.944803 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:20.944782    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54_odh-ai-gateway-infra(beecd43d-55b7-4310-8d29-3d7db1d92847)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847"
Nov 02 18:15:20.960651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.960624    5042 scope.go:122] "RemoveContainer" containerID="c55005ade233cff5eac6b0dfadcb2d7d415e7a1165886f0322e9a798bc246f4c"
Nov 02 18:15:20.983603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.982997    5042 scope.go:122] "RemoveContainer" containerID="2245714342e7b1fec4a9b2094465f9a9095c5e9adacc371395af88e2d0020037"
Nov 02 18:15:20.999540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:20.999508    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"]
Nov 02 18:15:21.006308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.006280    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-47e1c2-29894055-n4rlk"]
Nov 02 18:15:21.546981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.546946    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" path="/var/lib/kubelet/pods/0d7b295a-1204-4db7-81dd-bb7483eff02d/volumes"
Nov 02 18:15:21.960592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.960565    5042 generic.go:356] "Generic (PLEG): container finished" podID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerID="e36e8de50f98017e00503956df40092def6083a31e34dede837cd45b639f2a66" exitCode=60
Nov 02 18:15:21.960988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.960670    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" event={"ID":"40cccb0b-92ab-4399-a79d-65c5bc253aa0","Type":"ContainerDied","Data":"e36e8de50f98017e00503956df40092def6083a31e34dede837cd45b639f2a66"}
Nov 02 18:15:21.960988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.960732    5042 scope.go:122] "RemoveContainer" containerID="c559d5926cc408961cf933c7b83c2cc0a523eae6a829cb84cd4503df3e430f9f"
Nov 02 18:15:21.961270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.961247    5042 scope.go:122] "RemoveContainer" containerID="e36e8de50f98017e00503956df40092def6083a31e34dede837cd45b639f2a66"
Nov 02 18:15:21.961721 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:21.961698    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v_odh-ai-gateway-infra(40cccb0b-92ab-4399-a79d-65c5bc253aa0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0"
Nov 02 18:15:21.963167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.963144    5042 generic.go:356] "Generic (PLEG): container finished" podID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerID="6265cd8ee15c60ef503e82e754ed6979830194a6b3a5ce74ab6b83c28d7ebc1b" exitCode=60
Nov 02 18:15:21.963281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.963193    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" event={"ID":"ce6bc700-de9b-4e95-8d3a-a704283cd4fc","Type":"ContainerDied","Data":"6265cd8ee15c60ef503e82e754ed6979830194a6b3a5ce74ab6b83c28d7ebc1b"}
Nov 02 18:15:21.963686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.963668    5042 scope.go:122] "RemoveContainer" containerID="6265cd8ee15c60ef503e82e754ed6979830194a6b3a5ce74ab6b83c28d7ebc1b"
Nov 02 18:15:21.964138 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:21.964117    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6_odh-ai-gateway-infra(ce6bc700-de9b-4e95-8d3a-a704283cd4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc"
Nov 02 18:15:21.981935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:21.981909    5042 scope.go:122] "RemoveContainer" containerID="f9de8a602c8c65ee56ddf1f798acd547eccd987f92862c0f5e042698b0b98427"
Nov 02 18:15:22.012658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.011391    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"]
Nov 02 18:15:22.249491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.249462    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:22.354176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.354140    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:22.382388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.379515    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz\" (UniqueName: \"kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz\") pod \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") "
Nov 02 18:15:22.382388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.379642    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca\") pod \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\" (UID: \"2f9eff2e-1e2a-4472-b382-46a8fa8a995e\") "
Nov 02 18:15:22.382388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.380994    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca" pod "2f9eff2e-1e2a-4472-b382-46a8fa8a995e" (UID: "2f9eff2e-1e2a-4472-b382-46a8fa8a995e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:22.396911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.394037    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz" pod "2f9eff2e-1e2a-4472-b382-46a8fa8a995e" (UID: "2f9eff2e-1e2a-4472-b382-46a8fa8a995e"). InnerVolumeSpecName "kube-api-access-xrwbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:22.483212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.482136    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca\") pod \"beecd43d-55b7-4310-8d29-3d7db1d92847\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") "
Nov 02 18:15:22.483212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.482340    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v\" (UniqueName: \"kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v\") pod \"beecd43d-55b7-4310-8d29-3d7db1d92847\" (UID: \"beecd43d-55b7-4310-8d29-3d7db1d92847\") "
Nov 02 18:15:22.483212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.482659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca" pod "beecd43d-55b7-4310-8d29-3d7db1d92847" (UID: "beecd43d-55b7-4310-8d29-3d7db1d92847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:22.483780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.483273    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrwbz\" (UniqueName: \"kubernetes.io/projected/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-kube-api-access-xrwbz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:22.483780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.483294    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9eff2e-1e2a-4472-b382-46a8fa8a995e-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:22.483780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.483309    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beecd43d-55b7-4310-8d29-3d7db1d92847-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:22.485436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.485392    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v" pod "beecd43d-55b7-4310-8d29-3d7db1d92847" (UID: "beecd43d-55b7-4310-8d29-3d7db1d92847"). InnerVolumeSpecName "kube-api-access-x8z5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:22.537101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.537064    5042 scope.go:122] "RemoveContainer" containerID="3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52"
Nov 02 18:15:22.537435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.537403    5042 scope.go:122] "RemoveContainer" containerID="1f6ebb0c67e885e6c640a54389511b3d369d54c45dfdc28cf0f68f2827fca00f"
Nov 02 18:15:22.537552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.537530    5042 scope.go:122] "RemoveContainer" containerID="a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd"
Nov 02 18:15:22.638194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:22.610079    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8z5v\" (UniqueName: \"kubernetes.io/projected/beecd43d-55b7-4310-8d29-3d7db1d92847-kube-api-access-x8z5v\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:23.018592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.018561    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"
Nov 02 18:15:23.022668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.022386    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54" event={"ID":"beecd43d-55b7-4310-8d29-3d7db1d92847","Type":"ContainerDied","Data":"db707eb03ff5e363290bd326e3ae1fda176c45d3d0bfeb90fb78a1ea86e73536"}
Nov 02 18:15:23.022668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.022519    5042 scope.go:122] "RemoveContainer" containerID="d6379936c1ad1655384f8afa368dd5f9377e001ad93154dfc7fdaa3afde13380"
Nov 02 18:15:23.032913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.028809    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c" event={"ID":"2f9eff2e-1e2a-4472-b382-46a8fa8a995e","Type":"ContainerDied","Data":"a380acccfcfe11f1175eb1966ddcb4b5c33aac224b04f25df454f8cfd4b8c5e9"}
Nov 02 18:15:23.033137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.029660    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"
Nov 02 18:15:23.042948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.042922    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"]
Nov 02 18:15:23.059738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.059258    5042 scope.go:122] "RemoveContainer" containerID="be2b613ff5c473ffbb676ed819892c7de756be116d484f7e2331e09d1d5f4a07"
Nov 02 18:15:23.062450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.062417    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"]
Nov 02 18:15:23.118174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.118144    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"]
Nov 02 18:15:23.131991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.131956    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-72f605-29894055-zfr54"]
Nov 02 18:15:23.139016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.138986    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"]
Nov 02 18:15:23.158563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.148239    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c3c1f-29894055-8j97c"]
Nov 02 18:15:23.413955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.405489    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:23.590914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.581169    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv\" (UniqueName: \"kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv\") pod \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") "
Nov 02 18:15:23.590914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.581264    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca\") pod \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\" (UID: \"40cccb0b-92ab-4399-a79d-65c5bc253aa0\") "
Nov 02 18:15:23.601162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.598885    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca" pod "40cccb0b-92ab-4399-a79d-65c5bc253aa0" (UID: "40cccb0b-92ab-4399-a79d-65c5bc253aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:23.601162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.600166    5042 scope.go:122] "RemoveContainer" containerID="6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac"
Nov 02 18:15:23.601162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.601028    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40cccb0b-92ab-4399-a79d-65c5bc253aa0-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:23.603733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.601444    5042 scope.go:122] "RemoveContainer" containerID="415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6"
Nov 02 18:15:23.617442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.616409    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" path="/var/lib/kubelet/pods/2f9eff2e-1e2a-4472-b382-46a8fa8a995e/volumes"
Nov 02 18:15:23.617543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.617515    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" path="/var/lib/kubelet/pods/beecd43d-55b7-4310-8d29-3d7db1d92847/volumes"
Nov 02 18:15:23.638407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.638340    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv" pod "40cccb0b-92ab-4399-a79d-65c5bc253aa0" (UID: "40cccb0b-92ab-4399-a79d-65c5bc253aa0"). InnerVolumeSpecName "kube-api-access-rsmtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:23.696872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.696494    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:23.703428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.703393    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsmtv\" (UniqueName: \"kubernetes.io/projected/40cccb0b-92ab-4399-a79d-65c5bc253aa0-kube-api-access-rsmtv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:23.804221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.804192    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr\" (UniqueName: \"kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr\") pod \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") "
Nov 02 18:15:23.804686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.804665    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca\") pod \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\" (UID: \"ce6bc700-de9b-4e95-8d3a-a704283cd4fc\") "
Nov 02 18:15:23.806305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.806278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca" pod "ce6bc700-de9b-4e95-8d3a-a704283cd4fc" (UID: "ce6bc700-de9b-4e95-8d3a-a704283cd4fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:23.813062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.813022    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr" pod "ce6bc700-de9b-4e95-8d3a-a704283cd4fc" (UID: "ce6bc700-de9b-4e95-8d3a-a704283cd4fc"). InnerVolumeSpecName "kube-api-access-mschr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:23.909762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.909725    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:23.909762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:23.909765    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mschr\" (UniqueName: \"kubernetes.io/projected/ce6bc700-de9b-4e95-8d3a-a704283cd4fc-kube-api-access-mschr\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:24.041913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.041886    5042 generic.go:356] "Generic (PLEG): container finished" podID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerID="597d73f19595847d57b2d9e4bca1bb47dd685d4ae4de8612bbec8bb359e1dba5" exitCode=60
Nov 02 18:15:24.042266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.041933    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerDied","Data":"597d73f19595847d57b2d9e4bca1bb47dd685d4ae4de8612bbec8bb359e1dba5"}
Nov 02 18:15:24.042266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.041998    5042 scope.go:122] "RemoveContainer" containerID="a3c1c45e6de19b2efb9c644ae2ea028c0fe0e8ece5c629707d856c417e4866fd"
Nov 02 18:15:24.042453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.042436    5042 scope.go:122] "RemoveContainer" containerID="597d73f19595847d57b2d9e4bca1bb47dd685d4ae4de8612bbec8bb359e1dba5"
Nov 02 18:15:24.042880 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:24.042854    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm_odh-ai-gateway-infra(f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf"
Nov 02 18:15:24.045914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.045888    5042 generic.go:356] "Generic (PLEG): container finished" podID="7e6fff1c-409a-44c2-b849-63794da82212" containerID="bada037e0ea36293570a97d543d6063173d0de8fee94f8428a408402d48387b1" exitCode=60
Nov 02 18:15:24.046032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.045978    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" event={"ID":"7e6fff1c-409a-44c2-b849-63794da82212","Type":"ContainerDied","Data":"bada037e0ea36293570a97d543d6063173d0de8fee94f8428a408402d48387b1"}
Nov 02 18:15:24.046398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.046377    5042 scope.go:122] "RemoveContainer" containerID="bada037e0ea36293570a97d543d6063173d0de8fee94f8428a408402d48387b1"
Nov 02 18:15:24.046781 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:24.046758    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp_odh-ai-gateway-infra(7e6fff1c-409a-44c2-b849-63794da82212)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" podUID="7e6fff1c-409a-44c2-b849-63794da82212"
Nov 02 18:15:24.048357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.048340    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"
Nov 02 18:15:24.048488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.048368    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v" event={"ID":"40cccb0b-92ab-4399-a79d-65c5bc253aa0","Type":"ContainerDied","Data":"784c488b64287e0799bc724784d42670b638557a6303b4e2aaf3a118ad0d1adc"}
Nov 02 18:15:24.050642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.050622    5042 generic.go:356] "Generic (PLEG): container finished" podID="8431f13a-72b7-418f-8563-10812bf2c419" containerID="95f3bf4248d7f0ceefcf5b294bac97545d961cbf4e91d3cb28de42c0b8ddaca8" exitCode=60
Nov 02 18:15:24.050733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.050705    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" event={"ID":"8431f13a-72b7-418f-8563-10812bf2c419","Type":"ContainerDied","Data":"95f3bf4248d7f0ceefcf5b294bac97545d961cbf4e91d3cb28de42c0b8ddaca8"}
Nov 02 18:15:24.051183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.051163    5042 scope.go:122] "RemoveContainer" containerID="95f3bf4248d7f0ceefcf5b294bac97545d961cbf4e91d3cb28de42c0b8ddaca8"
Nov 02 18:15:24.051572 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:24.051545    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx_odh-ai-gateway-infra(8431f13a-72b7-418f-8563-10812bf2c419)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" podUID="8431f13a-72b7-418f-8563-10812bf2c419"
Nov 02 18:15:24.052353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.052331    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"
Nov 02 18:15:24.052435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.052365    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6" event={"ID":"ce6bc700-de9b-4e95-8d3a-a704283cd4fc","Type":"ContainerDied","Data":"72dde2415fb8ecc4c7669470d08f768580a8675b4f6c9dcdeb1956f1c79032a8"}
Nov 02 18:15:24.066545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.066527    5042 scope.go:122] "RemoveContainer" containerID="3556e581422ec59c4b61138f758c620e93ac7ca4a67aab0487eb77ac825f9b52"
Nov 02 18:15:24.101612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.101580    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"]
Nov 02 18:15:24.106527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.106500    5042 scope.go:122] "RemoveContainer" containerID="e36e8de50f98017e00503956df40092def6083a31e34dede837cd45b639f2a66"
Nov 02 18:15:24.109536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.109133    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-327608-29894055-mbb8v"]
Nov 02 18:15:24.122677 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.122652    5042 scope.go:122] "RemoveContainer" containerID="1f6ebb0c67e885e6c640a54389511b3d369d54c45dfdc28cf0f68f2827fca00f"
Nov 02 18:15:24.142316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.142283    5042 scope.go:122] "RemoveContainer" containerID="6265cd8ee15c60ef503e82e754ed6979830194a6b3a5ce74ab6b83c28d7ebc1b"
Nov 02 18:15:24.144099 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.144078    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"]
Nov 02 18:15:24.155342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.155317    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2db162-29894055-bv6w6"]
Nov 02 18:15:24.536217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.536180    5042 scope.go:122] "RemoveContainer" containerID="646da653fbafb373b7fe60e17ca02ad58ff26f317d5df3d3b3a04fb7ae0ff809"
Nov 02 18:15:24.536419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:24.536385    5042 scope.go:122] "RemoveContainer" containerID="254f7c3011273915825c3c8bffd3a9b972128cbfa4fb4848eba382a673deeb98"
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.077404    5042 generic.go:356] "Generic (PLEG): container finished" podID="3baacd62-a713-4016-850a-87cd47a0aa89" containerID="f816c442240c5475c585df199efc8bb7abcebdf01ed8e1501cc2b8d6a8c73072" exitCode=60
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.077953    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerDied","Data":"f816c442240c5475c585df199efc8bb7abcebdf01ed8e1501cc2b8d6a8c73072"}
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.078050    5042 scope.go:122] "RemoveContainer" containerID="6958d1151c44815eec2d7b7a5d9056bfb05e0d0c284d6c9a55f2ddf9dbfb19ac"
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.078707    5042 scope.go:122] "RemoveContainer" containerID="f816c442240c5475c585df199efc8bb7abcebdf01ed8e1501cc2b8d6a8c73072"
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:25.079218    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-sjzz6_odh-ai-gateway-infra(3baacd62-a713-4016-850a-87cd47a0aa89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" podUID="3baacd62-a713-4016-850a-87cd47a0aa89"
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.081218    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"]
Nov 02 18:15:25.097068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.085514    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerStarted","Data":"8d04ccc98b560ae70828305ec374622f6280465a976306db1307df197cebe931"}
Nov 02 18:15:25.129970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.129942    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"]
Nov 02 18:15:25.151410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:25.151371    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"]
Nov 02 18:15:26.278382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.278345    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:26.279371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.279323    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" path="/var/lib/kubelet/pods/40cccb0b-92ab-4399-a79d-65c5bc253aa0/volumes"
Nov 02 18:15:26.281027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.280746    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" path="/var/lib/kubelet/pods/ce6bc700-de9b-4e95-8d3a-a704283cd4fc/volumes"
Nov 02 18:15:26.283272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.283243    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"]
Nov 02 18:15:26.283402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.283312    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm" event={"ID":"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf","Type":"ContainerDied","Data":"b15a6dc6e2e305f8576ca11a5559ecab8d1bcbcdfb3d6823bf9d392fb33f958f"}
Nov 02 18:15:26.283402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.283349    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"]
Nov 02 18:15:26.283549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.283530    5042 scope.go:122] "RemoveContainer" containerID="597d73f19595847d57b2d9e4bca1bb47dd685d4ae4de8612bbec8bb359e1dba5"
Nov 02 18:15:26.288191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.288165    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:26.290077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.290046    5042 generic.go:356] "Generic (PLEG): container finished" podID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerID="8d04ccc98b560ae70828305ec374622f6280465a976306db1307df197cebe931" exitCode=60
Nov 02 18:15:26.290186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.290131    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerDied","Data":"8d04ccc98b560ae70828305ec374622f6280465a976306db1307df197cebe931"}
Nov 02 18:15:26.292994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.292967    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp" event={"ID":"7e6fff1c-409a-44c2-b849-63794da82212","Type":"ContainerDied","Data":"2bb366d4247281b99798c86c157686904225c085f2f4e5444c3aefff1137a035"}
Nov 02 18:15:26.293509 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.293487    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"
Nov 02 18:15:26.296993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.295581    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerStarted","Data":"b68312922af6b44069d316ff3ecb36bb9b2d46de4988e47e9f8c4444493cb623"}
Nov 02 18:15:26.305671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.305598    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54\" (UniqueName: \"kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54\") pod \"7e6fff1c-409a-44c2-b849-63794da82212\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") "
Nov 02 18:15:26.306159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.306125    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerStarted","Data":"8ccc136adc8851a1e8d95047c2b47c0d65ce7c3682bd967758f47c835cb3cd6d"}
Nov 02 18:15:26.307946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.307927    5042 scope.go:122] "RemoveContainer" containerID="415e0134633aed09c91ce317f4d74486db8dea436e26e52cc0272ba8429e7ca6"
Nov 02 18:15:26.312344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.312309    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54" pod "7e6fff1c-409a-44c2-b849-63794da82212" (UID: "7e6fff1c-409a-44c2-b849-63794da82212"). InnerVolumeSpecName "kube-api-access-wtd54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:26.376285 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.374819    5042 scope.go:122] "RemoveContainer" containerID="bada037e0ea36293570a97d543d6063173d0de8fee94f8428a408402d48387b1"
Nov 02 18:15:26.408785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.407387    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca\") pod \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") "
Nov 02 18:15:26.408785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.407664    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5\" (UniqueName: \"kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5\") pod \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\" (UID: \"f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf\") "
Nov 02 18:15:26.408785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.407819    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca\") pod \"7e6fff1c-409a-44c2-b849-63794da82212\" (UID: \"7e6fff1c-409a-44c2-b849-63794da82212\") "
Nov 02 18:15:26.408785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.408519    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtd54\" (UniqueName: \"kubernetes.io/projected/7e6fff1c-409a-44c2-b849-63794da82212-kube-api-access-wtd54\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.423278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.410355    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca" pod "f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" (UID: "f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:26.435325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.434066    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca" pod "7e6fff1c-409a-44c2-b849-63794da82212" (UID: "7e6fff1c-409a-44c2-b849-63794da82212"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:26.462556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.462521    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5" pod "f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" (UID: "f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf"). InnerVolumeSpecName "kube-api-access-kzgc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:26.466664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.466585    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:26.511566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.511524    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzgc5\" (UniqueName: \"kubernetes.io/projected/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-kube-api-access-kzgc5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.511792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.511776    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e6fff1c-409a-44c2-b849-63794da82212-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.511955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.511938    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.669694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.669604    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br\") pod \"8431f13a-72b7-418f-8563-10812bf2c419\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") "
Nov 02 18:15:26.669920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.669732    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca\") pod \"8431f13a-72b7-418f-8563-10812bf2c419\" (UID: \"8431f13a-72b7-418f-8563-10812bf2c419\") "
Nov 02 18:15:26.688883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.680707    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca" pod "8431f13a-72b7-418f-8563-10812bf2c419" (UID: "8431f13a-72b7-418f-8563-10812bf2c419"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:26.697678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.697639    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br" pod "8431f13a-72b7-418f-8563-10812bf2c419" (UID: "8431f13a-72b7-418f-8563-10812bf2c419"). InnerVolumeSpecName "kube-api-access-g45br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:26.727546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.727520    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:26.737980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.737944    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"]
Nov 02 18:15:26.743596 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.743565    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894055-t5bqp"]
Nov 02 18:15:26.767197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.767168    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:26.778496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.778467    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67\" (UniqueName: \"kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67\") pod \"3baacd62-a713-4016-850a-87cd47a0aa89\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") "
Nov 02 18:15:26.778741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.778708    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb\" (UniqueName: \"kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb\") pod \"f97c3798-d7b7-4b99-9d2c-3364239923aa\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") "
Nov 02 18:15:26.779300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.779278    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca\") pod \"3baacd62-a713-4016-850a-87cd47a0aa89\" (UID: \"3baacd62-a713-4016-850a-87cd47a0aa89\") "
Nov 02 18:15:26.779716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.779693    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca" pod "3baacd62-a713-4016-850a-87cd47a0aa89" (UID: "3baacd62-a713-4016-850a-87cd47a0aa89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:26.780138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.780115    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca\") pod \"f97c3798-d7b7-4b99-9d2c-3364239923aa\" (UID: \"f97c3798-d7b7-4b99-9d2c-3364239923aa\") "
Nov 02 18:15:26.780581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.780557    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca" pod "f97c3798-d7b7-4b99-9d2c-3364239923aa" (UID: "f97c3798-d7b7-4b99-9d2c-3364239923aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:26.781541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.781511    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97c3798-d7b7-4b99-9d2c-3364239923aa-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.781623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.781544    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45br\" (UniqueName: \"kubernetes.io/projected/8431f13a-72b7-418f-8563-10812bf2c419-kube-api-access-g45br\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.781623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.781561    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8431f13a-72b7-418f-8563-10812bf2c419-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.781623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.781576    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3baacd62-a713-4016-850a-87cd47a0aa89-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.783967 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.783939    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb" pod "f97c3798-d7b7-4b99-9d2c-3364239923aa" (UID: "f97c3798-d7b7-4b99-9d2c-3364239923aa"). InnerVolumeSpecName "kube-api-access-9smhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:26.797251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.797214    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67" pod "3baacd62-a713-4016-850a-87cd47a0aa89" (UID: "3baacd62-a713-4016-850a-87cd47a0aa89"). InnerVolumeSpecName "kube-api-access-q2q67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:26.883063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.883029    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2q67\" (UniqueName: \"kubernetes.io/projected/3baacd62-a713-4016-850a-87cd47a0aa89-kube-api-access-q2q67\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:26.883063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:26.883061    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smhb\" (UniqueName: \"kubernetes.io/projected/f97c3798-d7b7-4b99-9d2c-3364239923aa-kube-api-access-9smhb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:27.381679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.381643    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"]
Nov 02 18:15:27.450813 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.450773    5042 generic.go:356] "Generic (PLEG): container finished" podID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerID="b68312922af6b44069d316ff3ecb36bb9b2d46de4988e47e9f8c4444493cb623" exitCode=60
Nov 02 18:15:27.451090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.451061    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerDied","Data":"b68312922af6b44069d316ff3ecb36bb9b2d46de4988e47e9f8c4444493cb623"}
Nov 02 18:15:27.451164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.451154    5042 scope.go:122] "RemoveContainer" containerID="646da653fbafb373b7fe60e17ca02ad58ff26f317d5df3d3b3a04fb7ae0ff809"
Nov 02 18:15:27.457271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.457246    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"
Nov 02 18:15:27.460041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.458500    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx" event={"ID":"8431f13a-72b7-418f-8563-10812bf2c419","Type":"ContainerDied","Data":"3f5a33a7b35f4b06fd9825d2df917732f46c34963f688ded222c6b7f5abd4407"}
Nov 02 18:15:27.469767 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.469196    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"]
Nov 02 18:15:27.484610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.484404    5042 generic.go:356] "Generic (PLEG): container finished" podID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerID="8ccc136adc8851a1e8d95047c2b47c0d65ce7c3682bd967758f47c835cb3cd6d" exitCode=60
Nov 02 18:15:27.484804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.484682    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerDied","Data":"8ccc136adc8851a1e8d95047c2b47c0d65ce7c3682bd967758f47c835cb3cd6d"}
Nov 02 18:15:27.497040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.496104    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"
Nov 02 18:15:27.505799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.505765    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6" event={"ID":"3baacd62-a713-4016-850a-87cd47a0aa89","Type":"ContainerDied","Data":"10fe12b1003229239c5adcce9c1785af09409ce211eb202eadedc35ff897677a"}
Nov 02 18:15:27.506141 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.506123    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"
Nov 02 18:15:27.518638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.516143    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg" event={"ID":"f97c3798-d7b7-4b99-9d2c-3364239923aa","Type":"ContainerDied","Data":"7cc29d86bddedc45ab4ecb70203ad79dbdc3a03b73d9f0d668735ee67fd74b0c"}
Nov 02 18:15:27.518638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.516248    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"
Nov 02 18:15:27.538251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.538223    5042 scope.go:122] "RemoveContainer" containerID="95f3bf4248d7f0ceefcf5b294bac97545d961cbf4e91d3cb28de42c0b8ddaca8"
Nov 02 18:15:27.723880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:27.722757    5042 scope.go:122] "RemoveContainer" containerID="254f7c3011273915825c3c8bffd3a9b972128cbfa4fb4848eba382a673deeb98"
Nov 02 18:15:28.200016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.199992    5042 scope.go:122] "RemoveContainer" containerID="f816c442240c5475c585df199efc8bb7abcebdf01ed8e1501cc2b8d6a8c73072"
Nov 02 18:15:28.267966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.239551    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6fff1c-409a-44c2-b849-63794da82212" path="/var/lib/kubelet/pods/7e6fff1c-409a-44c2-b849-63794da82212/volumes"
Nov 02 18:15:28.291285 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.291250    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"]
Nov 02 18:15:28.297795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.297063    5042 scope.go:122] "RemoveContainer" containerID="8d04ccc98b560ae70828305ec374622f6280465a976306db1307df197cebe931"
Nov 02 18:15:28.303942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.301601    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-42d786-29894055-j9vnm"]
Nov 02 18:15:28.319691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.319638    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"]
Nov 02 18:15:28.343774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.343639    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-sjzz6"]
Nov 02 18:15:28.350403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.350377    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"]
Nov 02 18:15:28.354885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.354814    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:28.355895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.355740    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-807c08-29894055-kk2tg"]
Nov 02 18:15:28.362823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.362801    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"]
Nov 02 18:15:28.373913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.373878    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5ec436-29894055-tv8vx"]
Nov 02 18:15:28.418624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.418596    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:28.511647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.481443    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv\" (UniqueName: \"kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv\") pod \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") "
Nov 02 18:15:28.512005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.511974    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca\") pod \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\" (UID: \"b940ad60-3ce2-4bc7-8fa3-2346718ee228\") "
Nov 02 18:15:28.515630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.515049    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca" pod "b940ad60-3ce2-4bc7-8fa3-2346718ee228" (UID: "b940ad60-3ce2-4bc7-8fa3-2346718ee228"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:28.546366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.534529    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv" pod "b940ad60-3ce2-4bc7-8fa3-2346718ee228" (UID: "b940ad60-3ce2-4bc7-8fa3-2346718ee228"). InnerVolumeSpecName "kube-api-access-bq9cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:28.573614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.573580    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2" event={"ID":"b940ad60-3ce2-4bc7-8fa3-2346718ee228","Type":"ContainerDied","Data":"9d17bb4cbb72f23e051975a1f0f1dfd8f6186242cb5f67301cbd2097f9a51486"}
Nov 02 18:15:28.573818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.573637    5042 scope.go:122] "RemoveContainer" containerID="b68312922af6b44069d316ff3ecb36bb9b2d46de4988e47e9f8c4444493cb623"
Nov 02 18:15:28.573818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.573713    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"
Nov 02 18:15:28.579250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.579224    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4" event={"ID":"92658c40-3651-4c67-a106-b0ed6b13fb9e","Type":"ContainerDied","Data":"77f6aba08b48e78a298ad888e6397497fe0740e4710160250c2cb8b2fade9069"}
Nov 02 18:15:28.579455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.579434    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"
Nov 02 18:15:28.595486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.595463    5042 scope.go:122] "RemoveContainer" containerID="8ccc136adc8851a1e8d95047c2b47c0d65ce7c3682bd967758f47c835cb3cd6d"
Nov 02 18:15:28.614773 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.614736    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca\") pod \"92658c40-3651-4c67-a106-b0ed6b13fb9e\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") "
Nov 02 18:15:28.614980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.614962    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf\" (UniqueName: \"kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf\") pod \"92658c40-3651-4c67-a106-b0ed6b13fb9e\" (UID: \"92658c40-3651-4c67-a106-b0ed6b13fb9e\") "
Nov 02 18:15:28.616025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.616000    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9cv\" (UniqueName: \"kubernetes.io/projected/b940ad60-3ce2-4bc7-8fa3-2346718ee228-kube-api-access-bq9cv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:28.616116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.616032    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b940ad60-3ce2-4bc7-8fa3-2346718ee228-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:28.616774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.616648    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca" pod "92658c40-3651-4c67-a106-b0ed6b13fb9e" (UID: "92658c40-3651-4c67-a106-b0ed6b13fb9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:28.617509 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.617488    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"]
Nov 02 18:15:28.622880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.622858    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-24cfb6-29894055-lrpr2"]
Nov 02 18:15:28.626017 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.625990    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf" pod "92658c40-3651-4c67-a106-b0ed6b13fb9e" (UID: "92658c40-3651-4c67-a106-b0ed6b13fb9e"). InnerVolumeSpecName "kube-api-access-xr4wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:28.717784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.717757    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr4wf\" (UniqueName: \"kubernetes.io/projected/92658c40-3651-4c67-a106-b0ed6b13fb9e-kube-api-access-xr4wf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:28.717784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.717782    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92658c40-3651-4c67-a106-b0ed6b13fb9e-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:28.779958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.779862    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:15:28.912475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.912441    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"]
Nov 02 18:15:28.917477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:28.917446    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bee8f6-29894055-8sfd4"]
Nov 02 18:15:29.545931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:29.545895    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" path="/var/lib/kubelet/pods/3baacd62-a713-4016-850a-87cd47a0aa89/volumes"
Nov 02 18:15:29.546410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:29.546395    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8431f13a-72b7-418f-8563-10812bf2c419" path="/var/lib/kubelet/pods/8431f13a-72b7-418f-8563-10812bf2c419/volumes"
Nov 02 18:15:29.546795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:29.546782    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" path="/var/lib/kubelet/pods/92658c40-3651-4c67-a106-b0ed6b13fb9e/volumes"
Nov 02 18:15:29.547501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:29.547483    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" path="/var/lib/kubelet/pods/b940ad60-3ce2-4bc7-8fa3-2346718ee228/volumes"
Nov 02 18:15:29.547942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:29.547918    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" path="/var/lib/kubelet/pods/f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf/volumes"
Nov 02 18:15:29.548320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:29.548306    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" path="/var/lib/kubelet/pods/f97c3798-d7b7-4b99-9d2c-3364239923aa/volumes"
Nov 02 18:15:30.240314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:30.238090    5042 scope.go:122] "RemoveContainer" containerID="b160b69a14c8cf90daa6ed834e9b8a9e29a18291973756230eefdaa3389ec84e"
Nov 02 18:15:30.259654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:30.259626    5042 scope.go:122] "RemoveContainer" containerID="db368686f993616661e4ee1173fdc095a19c43f8b6d4f70feaf7c81d57c067f4"
Nov 02 18:15:30.273055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:30.272975    5042 scope.go:122] "RemoveContainer" containerID="e69714141d5aea66433b475d7bd1c3da5d22268871bd7e051a01b1eed2c99296"
Nov 02 18:15:30.288235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:30.288211    5042 scope.go:122] "RemoveContainer" containerID="baba4c48286127e54e3b8f5f42f59a4418264b3461b103d1f8b7e92711ebf712"
Nov 02 18:15:32.615288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.615251    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"]
Nov 02 18:15:32.616129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616109    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:32.616129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616129    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616144    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616150    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616157    5042 cpu_manager.go:414] "containerMap: removing container" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616162    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616170    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616175    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616180    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616184    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616193    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616198    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616203    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616208    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616218    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616222    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616236    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616240    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616246    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616250    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616263    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616267    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616278    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616284    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616290    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616294    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616302    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616306    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616315    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.616294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616319    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616325    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616330    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616340    5042 cpu_manager.go:414] "containerMap: removing container" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616344    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616350    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616355    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616367    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616372    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616381    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616386    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616392    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616396    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616405    5042 cpu_manager.go:414] "containerMap: removing container" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616409    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616417    5042 cpu_manager.go:414] "containerMap: removing container" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616421    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616428    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616432    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616440    5042 cpu_manager.go:414] "containerMap: removing container" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616444    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616451    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616455    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616464    5042 cpu_manager.go:414] "containerMap: removing container" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616468    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616474    5042 cpu_manager.go:414] "containerMap: removing container" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616478    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616490    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616493    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616502    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616506    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616613    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616621    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616629    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616636    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616641    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616647    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616652    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616657    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616663    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616668    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616674    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616681    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616686    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616691    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616696    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97c3798-d7b7-4b99-9d2c-3364239923aa" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616703    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616707    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616713    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616717    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="beecd43d-55b7-4310-8d29-3d7db1d92847" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616723    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6fff1c-409a-44c2-b849-63794da82212" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616728    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616734    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616741    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616748    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616752    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616758    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616764    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616772    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616777    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:32.617772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.616784    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:32.624953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.624927    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.630255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.630231    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"]
Nov 02 18:15:32.636271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.636248    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-22dfc1-dockercfg-rlrqw\""
Nov 02 18:15:32.636406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.636349    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-22dfc1\""
Nov 02 18:15:32.685454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.684805    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.685454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.685176    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dnx9\" (UniqueName: \"kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9\") pod \"payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.806872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.792560    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.806872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.792871    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dnx9\" (UniqueName: \"kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9\") pod \"payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.806872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.793757    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.858862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.851100    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dnx9\" (UniqueName: \"kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9\") pod \"payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") " pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:32.948877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:32.947442    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:33.611377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:33.611342    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"]
Nov 02 18:15:33.643618 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:33.643588    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" event={"ID":"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff","Type":"ContainerStarted","Data":"cf34e9bb9748705500dc4606d7a09e72f86588accfa55b747171a5407a6cd124"}
Nov 02 18:15:34.654872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:34.654812    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" event={"ID":"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff","Type":"ContainerStarted","Data":"6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e"}
Nov 02 18:15:34.675807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:34.675749    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" podStartSLOduration=2.675736478 podStartE2EDuration="2.675736478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:34.67154225 +0000 UTC m=+2547.250951639" watchObservedRunningTime="2026-11-02 18:15:34.675736478 +0000 UTC m=+2547.255145863"
Nov 02 18:15:35.666935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:35.666899    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:37.132592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.132553    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"]
Nov 02 18:15:37.135551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.135501    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:37.135738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.135722    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:37.135902 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.135890    5042 cpu_manager.go:414] "containerMap: removing container" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:37.136001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.135990    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92658c40-3651-4c67-a106-b0ed6b13fb9e" containerName="cleanup"
Nov 02 18:15:37.137414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137395    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:37.137574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137559    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:37.137663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137653    5042 cpu_manager.go:414] "containerMap: removing container" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:37.138320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137728    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cccb0b-92ab-4399-a79d-65c5bc253aa0" containerName="cleanup"
Nov 02 18:15:37.138320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137746    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:37.138320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137754    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:37.138320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137790    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:37.138320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.137798    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cfa5bc-4c9a-4fd8-8959-8332174a1aaf" containerName="cleanup"
Nov 02 18:15:37.138320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.138143    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6bc700-de9b-4e95-8d3a-a704283cd4fc" containerName="cleanup"
Nov 02 18:15:37.151263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.147634    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9eff2e-1e2a-4472-b382-46a8fa8a995e" containerName="cleanup"
Nov 02 18:15:37.151263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.147660    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3baacd62-a713-4016-850a-87cd47a0aa89" containerName="cleanup"
Nov 02 18:15:37.151263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.147687    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7b295a-1204-4db7-81dd-bb7483eff02d" containerName="cleanup"
Nov 02 18:15:37.151263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.147706    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b940ad60-3ce2-4bc7-8fa3-2346718ee228" containerName="cleanup"
Nov 02 18:15:37.151263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.147715    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8431f13a-72b7-418f-8563-10812bf2c419" containerName="cleanup"
Nov 02 18:15:37.154953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.154931    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"]
Nov 02 18:15:37.155947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.155345    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.170211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.168710    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-312f21-serving-cert\""
Nov 02 18:15:37.210339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.210288    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgzt\" (UniqueName: \"kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.210705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.210685    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.210947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.210926    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.211315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.211298    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.336933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.328673    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgzt\" (UniqueName: \"kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.336933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.328767    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.336933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.328827    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.336933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.329127    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.363393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.351887    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.375291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.375200    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.407053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.406762    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgzt\" (UniqueName: \"kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.424915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.424488    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls\") pod \"maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.493949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.493907    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:37.995448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:37.994614    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"]
Nov 02 18:15:38.923889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:38.923773    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf" event={"ID":"033e412d-b49f-4076-adec-af98cec94f3f","Type":"ContainerStarted","Data":"bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86"}
Nov 02 18:15:38.923889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:38.923885    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf" event={"ID":"033e412d-b49f-4076-adec-af98cec94f3f","Type":"ContainerStarted","Data":"185f6e02c254df6ccbcf310d7c806b97d933c374f57337ba656de681a547ea0f"}
Nov 02 18:15:38.924354 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:38.923914    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:38.945764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:38.945712    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf" podStartSLOduration=1.354982572 podStartE2EDuration="1.945697062s" totalImagesPullingTime="590.71449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:38.939679371 +0000 UTC m=+2551.519088785" watchObservedRunningTime="2026-11-02 18:15:38.945697062 +0000 UTC m=+2551.525106447"
Nov 02 18:15:40.320629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.320593    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"]
Nov 02 18:15:40.325542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.325523    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.329260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.329232    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:15:40.330392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.330365    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:15:40.333367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.333340    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"]
Nov 02 18:15:40.503652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.503616    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.504031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.504006    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.504358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.504341    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5thm\" (UniqueName: \"kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.606686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.606603    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.606977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.606948    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.607237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.607206    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5thm\" (UniqueName: \"kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.607501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.607473    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.609998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.609973    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.615965 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.615945    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5thm\" (UniqueName: \"kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm\") pod \"maas-api-revoke-sub-608e0df0-v25bq\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.657182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.657153    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:40.951945 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:40.951913    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda41bf0f_fc8c_4703_8c9b_b434990f054d.slice/crio-8e4468f3f5e5e6abdc25325ed5a55d9287a12e321fabff9fdafc3c72ad28b867 WatchSource:0}: Error finding container 8e4468f3f5e5e6abdc25325ed5a55d9287a12e321fabff9fdafc3c72ad28b867: Status 404 returned error can't find the container with id 8e4468f3f5e5e6abdc25325ed5a55d9287a12e321fabff9fdafc3c72ad28b867
Nov 02 18:15:40.953067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:40.953042    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"]
Nov 02 18:15:41.953964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:41.953919    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq" event={"ID":"da41bf0f-fc8c-4703-8c9b-b434990f054d","Type":"ContainerStarted","Data":"a0891be06fe62eee97af817e939e1fc526b7b9d7bea5223eef89aef7b94b22ed"}
Nov 02 18:15:41.954413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:41.953985    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq" event={"ID":"da41bf0f-fc8c-4703-8c9b-b434990f054d","Type":"ContainerStarted","Data":"8e4468f3f5e5e6abdc25325ed5a55d9287a12e321fabff9fdafc3c72ad28b867"}
Nov 02 18:15:41.978893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:41.974695    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq" podStartSLOduration=1.974676617 podStartE2EDuration="1.974676617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:41.970166802 +0000 UTC m=+2554.549576189" watchObservedRunningTime="2026-11-02 18:15:41.974676617 +0000 UTC m=+2554.554086002"
Nov 02 18:15:42.964492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:42.964459    5042 generic.go:356] "Generic (PLEG): container finished" podID="da41bf0f-fc8c-4703-8c9b-b434990f054d" containerID="a0891be06fe62eee97af817e939e1fc526b7b9d7bea5223eef89aef7b94b22ed" exitCode=0
Nov 02 18:15:42.964950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:42.964549    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq" event={"ID":"da41bf0f-fc8c-4703-8c9b-b434990f054d","Type":"ContainerDied","Data":"a0891be06fe62eee97af817e939e1fc526b7b9d7bea5223eef89aef7b94b22ed"}
Nov 02 18:15:44.222791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.222394    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:44.312663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.312567    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca\") pod \"da41bf0f-fc8c-4703-8c9b-b434990f054d\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") "
Nov 02 18:15:44.312663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.312613    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm\" (UniqueName: \"kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm\") pod \"da41bf0f-fc8c-4703-8c9b-b434990f054d\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") "
Nov 02 18:15:44.312935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.312684    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca\") pod \"da41bf0f-fc8c-4703-8c9b-b434990f054d\" (UID: \"da41bf0f-fc8c-4703-8c9b-b434990f054d\") "
Nov 02 18:15:44.313389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.313360    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca" pod "da41bf0f-fc8c-4703-8c9b-b434990f054d" (UID: "da41bf0f-fc8c-4703-8c9b-b434990f054d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:44.313891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.313867    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:44.315212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.315190    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca" pod "da41bf0f-fc8c-4703-8c9b-b434990f054d" (UID: "da41bf0f-fc8c-4703-8c9b-b434990f054d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:44.316453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.316429    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm" pod "da41bf0f-fc8c-4703-8c9b-b434990f054d" (UID: "da41bf0f-fc8c-4703-8c9b-b434990f054d"). InnerVolumeSpecName "kube-api-access-q5thm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:44.416166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.416136    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da41bf0f-fc8c-4703-8c9b-b434990f054d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:44.416166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.416164    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5thm\" (UniqueName: \"kubernetes.io/projected/da41bf0f-fc8c-4703-8c9b-b434990f054d-kube-api-access-q5thm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:44.939650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:44.939618    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:15:45.007637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:45.007593    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq" event={"ID":"da41bf0f-fc8c-4703-8c9b-b434990f054d","Type":"ContainerDied","Data":"8e4468f3f5e5e6abdc25325ed5a55d9287a12e321fabff9fdafc3c72ad28b867"}
Nov 02 18:15:45.007856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:45.007660    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e4468f3f5e5e6abdc25325ed5a55d9287a12e321fabff9fdafc3c72ad28b867"
Nov 02 18:15:45.007856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:45.007729    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"
Nov 02 18:15:46.035512 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:46.035462    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"]
Nov 02 18:15:46.040320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:46.040292    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a39c3cb-sm95j"]
Nov 02 18:15:46.732485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:46.732454    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:15:47.032779 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:47.032698    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"]
Nov 02 18:15:47.037775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:47.037749    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-879ffccf-mvrwt"]
Nov 02 18:15:47.545800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:47.545766    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90b1766-ee2e-4fc9-9f9c-b2f64704585e" path="/var/lib/kubelet/pods/e90b1766-ee2e-4fc9-9f9c-b2f64704585e/volumes"
Nov 02 18:15:47.547191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:47.547165    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89b4fb4-017a-4778-9ddd-92518d782dfb" path="/var/lib/kubelet/pods/f89b4fb4-017a-4778-9ddd-92518d782dfb/volumes"
Nov 02 18:15:48.634699 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:15:48.633353    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:15:50.815860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.815807    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"]
Nov 02 18:15:50.816727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.816709    5042 cpu_manager.go:414] "containerMap: removing container" podUID="da41bf0f-fc8c-4703-8c9b-b434990f054d" containerName="revoke-keys"
Nov 02 18:15:50.816780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.816730    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da41bf0f-fc8c-4703-8c9b-b434990f054d" containerName="revoke-keys"
Nov 02 18:15:50.816876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.816866    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="da41bf0f-fc8c-4703-8c9b-b434990f054d" containerName="revoke-keys"
Nov 02 18:15:50.822139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.822114    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:50.826275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.826252    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:15:50.826563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.826538    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:15:50.839879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.835508    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"]
Nov 02 18:15:50.986899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.986866    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:50.987090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.987062    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4gn\" (UniqueName: \"kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:50.987135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:50.987115    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.089129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.089044    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.089304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.089172    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4gn\" (UniqueName: \"kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.089304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.089208    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.089685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.089665    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.091821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.091794    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.097804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.097781    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4gn\" (UniqueName: \"kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn\") pod \"maas-api-revoke-sub-44364b1c-pcv9v\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.153098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.153063    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:51.543033 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:15:51.543000    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54a985f_9278_4a71_a016_807d2ecf6e19.slice/crio-49ee5ab48e546c21f5490d880f942a456bbbe2fe1a1c2fc886e1ab8406d22543 WatchSource:0}: Error finding container 49ee5ab48e546c21f5490d880f942a456bbbe2fe1a1c2fc886e1ab8406d22543: Status 404 returned error can't find the container with id 49ee5ab48e546c21f5490d880f942a456bbbe2fe1a1c2fc886e1ab8406d22543
Nov 02 18:15:51.550798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:51.550776    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"]
Nov 02 18:15:52.080177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:52.080137    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v" event={"ID":"c54a985f-9278-4a71-a016-807d2ecf6e19","Type":"ContainerStarted","Data":"49ee5ab48e546c21f5490d880f942a456bbbe2fe1a1c2fc886e1ab8406d22543"}
Nov 02 18:15:53.106108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:53.106012    5042 generic.go:356] "Generic (PLEG): container finished" podID="c54a985f-9278-4a71-a016-807d2ecf6e19" containerID="c4127b5cee27b284d04dbf8efca832ec64e2c79dd51caa2f85d4168cc5e3c444" exitCode=0
Nov 02 18:15:53.106539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:53.106171    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v" event={"ID":"c54a985f-9278-4a71-a016-807d2ecf6e19","Type":"ContainerDied","Data":"c4127b5cee27b284d04dbf8efca832ec64e2c79dd51caa2f85d4168cc5e3c444"}
Nov 02 18:15:54.326486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.326459    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:54.478638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.478600    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca\") pod \"c54a985f-9278-4a71-a016-807d2ecf6e19\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") "
Nov 02 18:15:54.478875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.478716    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn\" (UniqueName: \"kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn\") pod \"c54a985f-9278-4a71-a016-807d2ecf6e19\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") "
Nov 02 18:15:54.478875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.478823    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca\") pod \"c54a985f-9278-4a71-a016-807d2ecf6e19\" (UID: \"c54a985f-9278-4a71-a016-807d2ecf6e19\") "
Nov 02 18:15:54.479628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.479030    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca" pod "c54a985f-9278-4a71-a016-807d2ecf6e19" (UID: "c54a985f-9278-4a71-a016-807d2ecf6e19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:54.479628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.479598    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:54.481522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.481493    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn" pod "c54a985f-9278-4a71-a016-807d2ecf6e19" (UID: "c54a985f-9278-4a71-a016-807d2ecf6e19"). InnerVolumeSpecName "kube-api-access-js4gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:54.481658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.481555    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca" pod "c54a985f-9278-4a71-a016-807d2ecf6e19" (UID: "c54a985f-9278-4a71-a016-807d2ecf6e19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:54.585874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.585664    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4gn\" (UniqueName: \"kubernetes.io/projected/c54a985f-9278-4a71-a016-807d2ecf6e19-kube-api-access-js4gn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:54.585874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:54.585706    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c54a985f-9278-4a71-a016-807d2ecf6e19-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:15:55.125950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:55.125906    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"
Nov 02 18:15:55.126138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:55.125946    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v" event={"ID":"c54a985f-9278-4a71-a016-807d2ecf6e19","Type":"ContainerDied","Data":"49ee5ab48e546c21f5490d880f942a456bbbe2fe1a1c2fc886e1ab8406d22543"}
Nov 02 18:15:55.126138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:15:55.125996    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ee5ab48e546c21f5490d880f942a456bbbe2fe1a1c2fc886e1ab8406d22543"
Nov 02 18:16:01.057972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:01.057926    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"]
Nov 02 18:16:01.067334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:01.067288    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c547fc5c-r2xgv"]
Nov 02 18:16:01.586918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:01.583737    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66d1b9f-352f-40fa-af5c-bc04393d9e69" path="/var/lib/kubelet/pods/b66d1b9f-352f-40fa-af5c-bc04393d9e69/volumes"
Nov 02 18:16:05.280874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.278898    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"]
Nov 02 18:16:05.280874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.280409    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c54a985f-9278-4a71-a016-807d2ecf6e19" containerName="revoke-keys"
Nov 02 18:16:05.280874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.280440    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54a985f-9278-4a71-a016-807d2ecf6e19" containerName="revoke-keys"
Nov 02 18:16:05.280874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.280654    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54a985f-9278-4a71-a016-807d2ecf6e19" containerName="revoke-keys"
Nov 02 18:16:05.329209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.329174    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.352524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.341830    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"routing-overlay\""
Nov 02 18:16:05.352524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.343175    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-e2e--59b59218-dockercfg-lp4xp\""
Nov 02 18:16:05.352524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.343243    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"payload-processing-external-model-plugins-e2e-extmod-a-36a991\""
Nov 02 18:16:05.352524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.343340    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"openshift-service-ca.crt\""
Nov 02 18:16:05.352524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.343419    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"kube-root-ca.crt\""
Nov 02 18:16:05.352524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.343475    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-36a991\"/\"agc-cred-0ebc5b098a14\""
Nov 02 18:16:05.354206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.354169    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.354553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.354519    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.354762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.354742    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvljp\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.355029 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.355009    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.397863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.368663    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"]
Nov 02 18:16:05.679012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.477293    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvljp\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.689152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.576440    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvljp\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.827681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.720100    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.842897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.842858    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.860194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.860161    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.860615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.855882    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.860740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.804794    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:05.874295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:05.874258    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") " pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:06.020441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.020376    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:06.025997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.025907    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 18:16:06.091546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.091500    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"]
Nov 02 18:16:06.099578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.099547    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.102680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.102652    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"]
Nov 02 18:16:06.188916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.167234    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0b0610-dockercfg-v77zv\""
Nov 02 18:16:06.274798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.213362    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0b0610\""
Nov 02 18:16:06.277831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.277802    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.282717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.282671    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67c7\" (UniqueName: \"kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7\") pod \"payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.424088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.424051    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.424587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.424560    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67c7\" (UniqueName: \"kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7\") pod \"payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.428863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.425784    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.572463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.572382    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67c7\" (UniqueName: \"kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7\") pod \"payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.604990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.578171    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:06.675465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:06.634367    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"]
Nov 02 18:16:07.237895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:07.205389    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"]
Nov 02 18:16:07.237895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:07.215924    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:16:07.458079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:07.377540    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:16:08.041019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.040975    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:16:08.042217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.041381    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 18:16:08.043062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.042358    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 18:16:08.047805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.044709    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:16:08.048172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.048138    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" podUID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:16:08.057807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.057555    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:16:08.065949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.063550    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd" podUID="0b474fd7-9730-4acd-be17-6e9ae01a1e44" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:16:08.104016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.102592    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"]
Nov 02 18:16:08.343543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:08.343498    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"]
Nov 02 18:16:08.343684 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:16:08.343579    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c97ee6f_f862_41b0_84a7_6c264cdc3e37.slice/crio-86c8f0cf7a2adadce2588a9ccca5b93774a6ad06f03c5d08112373c5ff537509 WatchSource:0}: Error finding container 86c8f0cf7a2adadce2588a9ccca5b93774a6ad06f03c5d08112373c5ff537509: Status 404 returned error can't find the container with id 86c8f0cf7a2adadce2588a9ccca5b93774a6ad06f03c5d08112373c5ff537509
Nov 02 18:16:09.115274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.115228    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" event={"ID":"5c97ee6f-f862-41b0-84a7-6c264cdc3e37","Type":"ContainerStarted","Data":"a94f48dd752e7481b78be34df1fe6772e02e86d4494d2412e85400e7753b4aa5"}
Nov 02 18:16:09.115786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.115298    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" event={"ID":"5c97ee6f-f862-41b0-84a7-6c264cdc3e37","Type":"ContainerStarted","Data":"86c8f0cf7a2adadce2588a9ccca5b93774a6ad06f03c5d08112373c5ff537509"}
Nov 02 18:16:09.115786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.115326    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:09.117359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.117326    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl" event={"ID":"147c3af4-7b9e-415b-9d83-78e60ebdd602","Type":"ContainerStarted","Data":"045032b364b68f32cb62c405b6f303d30513c71338383e98806cad710c0707d5"}
Nov 02 18:16:09.117492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.117384    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl" event={"ID":"147c3af4-7b9e-415b-9d83-78e60ebdd602","Type":"ContainerStarted","Data":"eb8721a3d0a364ea7b9b50a02d511899a6e69f3f2e125b04380619dec569201e"}
Nov 02 18:16:09.117492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.117413    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:09.137612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.137565    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" podStartSLOduration=3.137550889 podStartE2EDuration="3.137550889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:09.132468346 +0000 UTC m=+2581.711877734" watchObservedRunningTime="2026-11-02 18:16:09.137550889 +0000 UTC m=+2581.716960274"
Nov 02 18:16:09.152406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.152355    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl" podStartSLOduration=4.152339568 podStartE2EDuration="4.152339568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:09.148349762 +0000 UTC m=+2581.727759150" watchObservedRunningTime="2026-11-02 18:16:09.152339568 +0000 UTC m=+2581.731748954"
Nov 02 18:16:09.887103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.887065    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"]
Nov 02 18:16:09.887303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.887271    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy" containerID="cri-o://f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef" gracePeriod=30
Nov 02 18:16:09.949881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.949826    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"]
Nov 02 18:16:09.950224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:09.950194    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy" containerID="cri-o://56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3" gracePeriod=30
Nov 02 18:16:12.222527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.222478    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx" podUID="aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" containerName="payload-processing" containerID="cri-o://af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11" gracePeriod=25
Nov 02 18:16:12.902900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.902769    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:16:12.945819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.945787    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2\") pod \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") "
Nov 02 18:16:12.946028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.945875    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume\") pod \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") "
Nov 02 18:16:12.946028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.945949    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume\") pod \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") "
Nov 02 18:16:12.946028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.946005    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials\") pod \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\" (UID: \"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3\") "
Nov 02 18:16:12.946379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.946316    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume" pod "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" (UID: "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:12.946474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.946424    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume" pod "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" (UID: "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:12.947050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.947023    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:12.947169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.947053    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:12.948817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.948779    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2" pod "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" (UID: "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3"). InnerVolumeSpecName "kube-api-access-2vvn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:12.949193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:12.949170    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials" pod "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" (UID: "aacc8ae3-f53c-4513-8b16-2b91dc00cdd3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:13.049045 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.048958    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vvn2\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-kube-api-access-2vvn2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:13.049045 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.048987    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:13.328390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.328292    5042 generic.go:356] "Generic (PLEG): container finished" podID="aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" containerID="af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11" exitCode=0
Nov 02 18:16:13.328390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.328335    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"
Nov 02 18:16:13.328390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.328367    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx" event={"ID":"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3","Type":"ContainerDied","Data":"af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11"}
Nov 02 18:16:13.328996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.328423    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx" event={"ID":"aacc8ae3-f53c-4513-8b16-2b91dc00cdd3","Type":"ContainerDied","Data":"4b8f049e128868f1e84d54a7df3b024ce6cfbb8af075b8531e1a86c8cc17ad1c"}
Nov 02 18:16:13.328996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.328446    5042 scope.go:122] "RemoveContainer" containerID="af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11"
Nov 02 18:16:13.343327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.343300    5042 scope.go:122] "RemoveContainer" containerID="af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11"
Nov 02 18:16:13.343659 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:16:13.343634    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11\": container with ID starting with af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11 not found: ID does not exist" containerID="af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11"
Nov 02 18:16:13.343755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.343674    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11"} err="failed to get container status \"af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11\": rpc error: code = NotFound desc = could not find container \"af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11\": container with ID starting with af9d47c1fd4ce89ba787f67534cdd5c9507c1645acf0664ad065c9d702328a11 not found: ID does not exist"
Nov 02 18:16:13.355170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.355135    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"]
Nov 02 18:16:13.360242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.360209    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2db162/payload-processing-external-model-e2e-extmod-b-2db162-74f4d2lnx"]
Nov 02 18:16:13.547223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:13.547188    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" path="/var/lib/kubelet/pods/aacc8ae3-f53c-4513-8b16-2b91dc00cdd3/volumes"
Nov 02 18:16:14.117660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:14.117556    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl" podUID="147c3af4-7b9e-415b-9d83-78e60ebdd602" containerName="payload-processing" containerID="cri-o://045032b364b68f32cb62c405b6f303d30513c71338383e98806cad710c0707d5" gracePeriod=25
Nov 02 18:16:14.842895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:14.842857    5042 generic.go:356] "Generic (PLEG): container finished" podID="147c3af4-7b9e-415b-9d83-78e60ebdd602" containerID="045032b364b68f32cb62c405b6f303d30513c71338383e98806cad710c0707d5" exitCode=0
Nov 02 18:16:14.859198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:14.856110    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl" event={"ID":"147c3af4-7b9e-415b-9d83-78e60ebdd602","Type":"ContainerDied","Data":"045032b364b68f32cb62c405b6f303d30513c71338383e98806cad710c0707d5"}
Nov 02 18:16:15.112326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.112295    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.227153    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp\") pod \"147c3af4-7b9e-415b-9d83-78e60ebdd602\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") "
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.227415    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume\") pod \"147c3af4-7b9e-415b-9d83-78e60ebdd602\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") "
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.227520    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume\") pod \"147c3af4-7b9e-415b-9d83-78e60ebdd602\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") "
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.227641    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials\") pod \"147c3af4-7b9e-415b-9d83-78e60ebdd602\" (UID: \"147c3af4-7b9e-415b-9d83-78e60ebdd602\") "
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.228203    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume" pod "147c3af4-7b9e-415b-9d83-78e60ebdd602" (UID: "147c3af4-7b9e-415b-9d83-78e60ebdd602"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.228278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume" pod "147c3af4-7b9e-415b-9d83-78e60ebdd602" (UID: "147c3af4-7b9e-415b-9d83-78e60ebdd602"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.228531    5042 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-routing-overlay-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.228621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.228551    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/147c3af4-7b9e-415b-9d83-78e60ebdd602-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.232890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.231359    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp" pod "147c3af4-7b9e-415b-9d83-78e60ebdd602" (UID: "147c3af4-7b9e-415b-9d83-78e60ebdd602"). InnerVolumeSpecName "kube-api-access-mvljp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:15.232890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.232025    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials" pod "147c3af4-7b9e-415b-9d83-78e60ebdd602" (UID: "147c3af4-7b9e-415b-9d83-78e60ebdd602"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:15.360709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.360670    5042 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-provider-credentials\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.360709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.360711    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvljp\" (UniqueName: \"kubernetes.io/projected/147c3af4-7b9e-415b-9d83-78e60ebdd602-kube-api-access-mvljp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.399227 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.399189    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 18:16:15.401826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.401802    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 18:16:15.564301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564272    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564313    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.564474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564336    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.564474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564395    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564421    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564526    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.564632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564547    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.564632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564575    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.564750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564666    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.564750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564689    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564728    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564751    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564785    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.564881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564802    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.565030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.564945    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.565096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.565036    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token\") pod \"04dbd50e-774a-48ff-9f8b-1337393e01a0\" (UID: \"04dbd50e-774a-48ff-9f8b-1337393e01a0\") "
Nov 02 18:16:15.565157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.565103    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.565157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.565132    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy\") pod \"9d557769-1af8-496c-80a5-c5bccf34f277\" (UID: \"9d557769-1af8-496c-80a5-c5bccf34f277\") "
Nov 02 18:16:15.566270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566130    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.566270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566223    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.566587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566264    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.566587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566465    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.567350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566667    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.567350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566689    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.567350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566704    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.567350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566719    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.567350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.566979    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:15.568177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.567620    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.568177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.567828    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.568177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.567833    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "kube-api-access-q2ptt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:15.568177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.568065    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.568177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.568139    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:16:15.568411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.568308    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.590452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.590393    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.599352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.599088    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:15.600300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.600258    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "kube-api-access-8j4lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:15.601275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.601238    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:16:15.602119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.602023    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:15.602119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.602087    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token" pod "04dbd50e-774a-48ff-9f8b-1337393e01a0" (UID: "04dbd50e-774a-48ff-9f8b-1337393e01a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:15.602661 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.602640    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo" pod "9d557769-1af8-496c-80a5-c5bccf34f277" (UID: "9d557769-1af8-496c-80a5-c5bccf34f277"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:16:15.669327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669267    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d557769-1af8-496c-80a5-c5bccf34f277-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669308    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669324    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669341    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04dbd50e-774a-48ff-9f8b-1337393e01a0-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669359    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669374    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669387    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669403    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669418    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669431    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d557769-1af8-496c-80a5-c5bccf34f277-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669445    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d557769-1af8-496c-80a5-c5bccf34f277-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669460    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04dbd50e-774a-48ff-9f8b-1337393e01a0-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669474    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ptt\" (UniqueName: \"kubernetes.io/projected/9d557769-1af8-496c-80a5-c5bccf34f277-kube-api-access-q2ptt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.669651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.669490    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j4lk\" (UniqueName: \"kubernetes.io/projected/04dbd50e-774a-48ff-9f8b-1337393e01a0-kube-api-access-8j4lk\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:16:15.876200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.876106    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"
Nov 02 18:16:15.876200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.876122    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl" event={"ID":"147c3af4-7b9e-415b-9d83-78e60ebdd602","Type":"ContainerDied","Data":"eb8721a3d0a364ea7b9b50a02d511899a6e69f3f2e125b04380619dec569201e"}
Nov 02 18:16:15.876752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.876237    5042 scope.go:122] "RemoveContainer" containerID="045032b364b68f32cb62c405b6f303d30513c71338383e98806cad710c0707d5"
Nov 02 18:16:15.878265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.878139    5042 generic.go:356] "Generic (PLEG): container finished" podID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerID="f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef" exitCode=0
Nov 02 18:16:15.878265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.878226    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"
Nov 02 18:16:15.878387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.878265    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" event={"ID":"04dbd50e-774a-48ff-9f8b-1337393e01a0","Type":"ContainerDied","Data":"f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef"}
Nov 02 18:16:15.878387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.878305    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw" event={"ID":"04dbd50e-774a-48ff-9f8b-1337393e01a0","Type":"ContainerDied","Data":"443637b023d197e5a14a3b80221d0d4fc3c26b48280da9b4764c7f2bf5a4ce59"}
Nov 02 18:16:15.881489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.881260    5042 generic.go:356] "Generic (PLEG): container finished" podID="9d557769-1af8-496c-80a5-c5bccf34f277" containerID="56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3" exitCode=0
Nov 02 18:16:15.881489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.881305    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" event={"ID":"9d557769-1af8-496c-80a5-c5bccf34f277","Type":"ContainerDied","Data":"56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3"}
Nov 02 18:16:15.881489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.881387    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh" event={"ID":"9d557769-1af8-496c-80a5-c5bccf34f277","Type":"ContainerDied","Data":"ec04e63777926c709d02fdef6e93d60b7d69d961e1c657d7f751f42838bb995b"}
Nov 02 18:16:15.881489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.881312    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"
Nov 02 18:16:15.890876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.890763    5042 scope.go:122] "RemoveContainer" containerID="f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef"
Nov 02 18:16:15.903370 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.903343    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"]
Nov 02 18:16:15.908286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.908261    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-36a991/payload-processing-external-model-e2e-extmod-a-36a991-d4577pfrl"]
Nov 02 18:16:15.911144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.911123    5042 scope.go:122] "RemoveContainer" containerID="f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef"
Nov 02 18:16:15.911701 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:16:15.911675    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef\": container with ID starting with f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef not found: ID does not exist" containerID="f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef"
Nov 02 18:16:15.911874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.911824    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef"} err="failed to get container status \"f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef\": rpc error: code = NotFound desc = could not find container \"f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef\": container with ID starting with f7bfcb3b2d5c7b7720625ddde4ef1c49eb3ac3372c3448a386501ab3de5682ef not found: ID does not exist"
Nov 02 18:16:15.911983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:15.911971    5042 scope.go:122] "RemoveContainer" containerID="56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3"
Nov 02 18:16:16.043947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:16.043326    5042 scope.go:122] "RemoveContainer" containerID="56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3"
Nov 02 18:16:16.043947 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:16:16.043783    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3\": container with ID starting with 56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3 not found: ID does not exist" containerID="56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3"
Nov 02 18:16:16.043947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:16.043830    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3"} err="failed to get container status \"56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3\": rpc error: code = NotFound desc = could not find container \"56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3\": container with ID starting with 56f08bca3446b4785ce7005b5748f778ddc2e8f1fad35c15a22d96a45ec509e3 not found: ID does not exist"
Nov 02 18:16:16.057875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:16.054946    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"]
Nov 02 18:16:16.062865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:16.061625    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-lllgh"]
Nov 02 18:16:16.067863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:16.066899    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"]
Nov 02 18:16:16.072934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:16.072351    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-bdmgw"]
Nov 02 18:16:17.546994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:17.546951    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" path="/var/lib/kubelet/pods/04dbd50e-774a-48ff-9f8b-1337393e01a0/volumes"
Nov 02 18:16:17.548023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:17.547636    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="147c3af4-7b9e-415b-9d83-78e60ebdd602" path="/var/lib/kubelet/pods/147c3af4-7b9e-415b-9d83-78e60ebdd602/volumes"
Nov 02 18:16:17.548407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:17.548387    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" path="/var/lib/kubelet/pods/9d557769-1af8-496c-80a5-c5bccf34f277/volumes"
Nov 02 18:16:18.680000 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:16:18.676454    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.924711    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"]
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.925967    5042 cpu_manager.go:414] "containerMap: removing container" podUID="147c3af4-7b9e-415b-9d83-78e60ebdd602" containerName="payload-processing"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.925990    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="147c3af4-7b9e-415b-9d83-78e60ebdd602" containerName="payload-processing"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926010    5042 cpu_manager.go:414] "containerMap: removing container" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926017    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926032    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926040    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926101    5042 cpu_manager.go:414] "containerMap: removing container" podUID="aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" containerName="payload-processing"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926109    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" containerName="payload-processing"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926399    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacc8ae3-f53c-4513-8b16-2b91dc00cdd3" containerName="payload-processing"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926430    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="147c3af4-7b9e-415b-9d83-78e60ebdd602" containerName="payload-processing"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926443    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="04dbd50e-774a-48ff-9f8b-1337393e01a0" containerName="istio-proxy"
Nov 02 18:16:19.929899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.926452    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d557769-1af8-496c-80a5-c5bccf34f277" containerName="istio-proxy"
Nov 02 18:16:19.986279 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.984877    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:19.996401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.994604    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b3a6\""
Nov 02 18:16:19.996401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:19.995427    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b3a6-dockercfg-q75fl\""
Nov 02 18:16:20.375230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.375197    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:16:20.453713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.453672    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"]
Nov 02 18:16:20.599865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.599303    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2cz\" (UniqueName: \"kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz\") pod \"payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:20.599865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.599425    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:20.704866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.702897    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2cz\" (UniqueName: \"kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz\") pod \"payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:20.704866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.703056    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:20.754800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.751907    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:20.759165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.759130    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2cz\" (UniqueName: \"kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz\") pod \"payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:20.798319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:20.794134    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:21.762681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:21.760618    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"]
Nov 02 18:16:22.541280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:22.541176    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" event={"ID":"72ee01e7-6cc0-4218-aa5d-65c64cfff50a","Type":"ContainerStarted","Data":"18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717"}
Nov 02 18:16:22.541620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:22.541594    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" event={"ID":"72ee01e7-6cc0-4218-aa5d-65c64cfff50a","Type":"ContainerStarted","Data":"ab8b7712b88cab9da29a9587d3657cc3bf47eef2482657584abfe5af18591edf"}
Nov 02 18:16:22.543145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:22.543122    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:22.599473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:22.599399    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" podStartSLOduration=3.599378928 podStartE2EDuration="3.599378928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:22.579997168 +0000 UTC m=+2595.159406554" watchObservedRunningTime="2026-11-02 18:16:22.599378928 +0000 UTC m=+2595.178788316"
Nov 02 18:16:27.886976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:27.886943    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"]
Nov 02 18:16:27.896565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:27.896536    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:27.911880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:27.911575    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-63be7a-openshift-default-dockercfg-5274g\""
Nov 02 18:16:27.952332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:27.952249    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"]
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.121664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.121904    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8nj\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122001    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122110    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122155    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122186    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122220    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122280    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.147921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.122310    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.330863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.270220    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.337104 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.337070    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.343028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.342996    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.343226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.343212    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.370867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.346982    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.371270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.371237    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.371419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.371404    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.371748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.371719    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.372159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.372134    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8nj\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.376861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.339298    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.456649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.345745    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.456649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.402740    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.456649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.350248    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.456649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.345426    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.456649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.417810    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.490864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.484183    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.500943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.497412    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.521067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.520999    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8nj\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.554326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.554273    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:28.944059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.943317    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"]
Nov 02 18:16:28.949025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.945111    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:16:28.949025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.945216    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:16:28.949025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.945274    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:16:28.962860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:28.960546    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" event={"ID":"19ba2ee9-18e8-4712-ad8b-da7052994a39","Type":"ContainerStarted","Data":"e0fd78d2350a035dc7f26d0e027a96665a5d1ccfeb52c6c4983038e9d6c47d29"}
Nov 02 18:16:29.972813 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:29.972774    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" event={"ID":"19ba2ee9-18e8-4712-ad8b-da7052994a39","Type":"ContainerStarted","Data":"9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"}
Nov 02 18:16:30.001019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.000955    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podStartSLOduration=3.000937787 podStartE2EDuration="3.000937787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:29.998554542 +0000 UTC m=+2602.577963926" watchObservedRunningTime="2026-11-02 18:16:30.000937787 +0000 UTC m=+2602.580347184"
Nov 02 18:16:30.483647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.483608    5042 scope.go:122] "RemoveContainer" containerID="fa00ea7366bcd1f5ed6a9e8e861a48cc715bffaf6bafa540c314e9590c260745"
Nov 02 18:16:30.555344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.555239    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:30.567383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.567349    5042 scope.go:122] "RemoveContainer" containerID="bf1913b1e945acd4ec5329d8aa4bc2e275d2169e39f5eada2c3f363c3383454d"
Nov 02 18:16:30.569991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.569970    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:30.583785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.583763    5042 scope.go:122] "RemoveContainer" containerID="059cbf6f2c1cf885645cdb09bee9f55a821e3b663291e0716e55b1ec14340630"
Nov 02 18:16:30.601929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.601813    5042 scope.go:122] "RemoveContainer" containerID="30b39e8cb5800a6639b6c3e217c84df745d84e2a40be5c529dbffc240c1a942a"
Nov 02 18:16:30.983143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.983111    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:30.984277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:30.984248    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:16:33.506991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.506959    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"]
Nov 02 18:16:33.511884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.511851    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:33.525817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.525668    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-63be7a\""
Nov 02 18:16:33.525817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.525769    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-63be7a-dockercfg-7xlt2\""
Nov 02 18:16:33.532003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.531907    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"]
Nov 02 18:16:33.552722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.552691    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"]
Nov 02 18:16:33.563666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.563629    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:33.573415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.573362    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"]
Nov 02 18:16:33.638512 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.638451    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:33.638825 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.638798    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z94t\" (UniqueName: \"kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t\") pod \"payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:33.639071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.639051    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:33.639389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.639369    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbrv\" (UniqueName: \"kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv\") pod \"payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:33.748885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.748819    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:33.785915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.785208    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z94t\" (UniqueName: \"kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t\") pod \"payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:33.785915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.785512    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:33.785915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.785746    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbrv\" (UniqueName: \"kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv\") pod \"payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:33.813865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.807003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:33.838867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:33.830663    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:34.150947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.150869    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbrv\" (UniqueName: \"kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv\") pod \"payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:34.152577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.151482    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z94t\" (UniqueName: \"kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t\") pod \"payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:34.207615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.207557    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:34.445995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.445962    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:34.510101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.509869    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"]
Nov 02 18:16:34.511855 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:16:34.511067    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256a31ec_4dd1_454e_b72b_474fcbf1a835.slice/crio-925aaded61b37d17a0c7b0b8c3ca1ddc4972754deae73be203d8562d91b149d0 WatchSource:0}: Error finding container 925aaded61b37d17a0c7b0b8c3ca1ddc4972754deae73be203d8562d91b149d0: Status 404 returned error can't find the container with id 925aaded61b37d17a0c7b0b8c3ca1ddc4972754deae73be203d8562d91b149d0
Nov 02 18:16:34.757706 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:16:34.757673    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d78a3b0_6db9_4a35_a271_04a31039461f.slice/crio-106a82cfa89ec003ff1ff377108bfab89b5cc26f2fb5f3c6babceab8b7a33190 WatchSource:0}: Error finding container 106a82cfa89ec003ff1ff377108bfab89b5cc26f2fb5f3c6babceab8b7a33190: Status 404 returned error can't find the container with id 106a82cfa89ec003ff1ff377108bfab89b5cc26f2fb5f3c6babceab8b7a33190
Nov 02 18:16:34.758478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.758453    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"]
Nov 02 18:16:34.850305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:34.850277    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:16:35.126119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.126021    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk" event={"ID":"256a31ec-4dd1-454e-b72b-474fcbf1a835","Type":"ContainerStarted","Data":"7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29"}
Nov 02 18:16:35.126119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.126082    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk" event={"ID":"256a31ec-4dd1-454e-b72b-474fcbf1a835","Type":"ContainerStarted","Data":"925aaded61b37d17a0c7b0b8c3ca1ddc4972754deae73be203d8562d91b149d0"}
Nov 02 18:16:35.126119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.126109    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:35.127895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.127826    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" event={"ID":"4d78a3b0-6db9-4a35-a271-04a31039461f","Type":"ContainerStarted","Data":"1d5b2d2dd57d0358a663d98b98bba5bccc6192f7c3b55eb737bdee6ab9392514"}
Nov 02 18:16:35.128041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.127914    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" event={"ID":"4d78a3b0-6db9-4a35-a271-04a31039461f","Type":"ContainerStarted","Data":"106a82cfa89ec003ff1ff377108bfab89b5cc26f2fb5f3c6babceab8b7a33190"}
Nov 02 18:16:35.128041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.127941    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:35.151332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.151276    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk" podStartSLOduration=2.151258708 podStartE2EDuration="2.151258708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:35.143387925 +0000 UTC m=+2607.722797311" watchObservedRunningTime="2026-11-02 18:16:35.151258708 +0000 UTC m=+2607.730668097"
Nov 02 18:16:35.168124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:35.168078    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" podStartSLOduration=2.168063717 podStartE2EDuration="2.168063717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:35.164787595 +0000 UTC m=+2607.744196990" watchObservedRunningTime="2026-11-02 18:16:35.168063717 +0000 UTC m=+2607.747473103"
Nov 02 18:16:46.140824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:46.140785    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:16:46.141395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:46.140919    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:16:47.911451 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:47.911415    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"]
Nov 02 18:16:47.918680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:47.918642    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:47.929901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:47.927084    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50b3a6-serving-cert\""
Nov 02 18:16:47.935290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:47.935263    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"]
Nov 02 18:16:48.204377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.201207    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkdj\" (UniqueName: \"kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.204377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.202240    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.204377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.202332    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.204377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.202480    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.304773 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.304724    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkdj\" (UniqueName: \"kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.305037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.305008    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.305110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.305079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.305214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.305197    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.306169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.306139    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.308124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.308102    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.308241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.308139    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.316503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.316474    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkdj\" (UniqueName: \"kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj\") pod \"maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.549068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:48.548982    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:48.736861 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:16:48.736468    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:16:49.102028 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:16:49.101995    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec5ab2d_3e3d_4f09_bc06_41e30894c7af.slice/crio-3cc601da5a8778a3701b8ed917a79540e0a11cf04ac05078ae4c517f65a115b6 WatchSource:0}: Error finding container 3cc601da5a8778a3701b8ed917a79540e0a11cf04ac05078ae4c517f65a115b6: Status 404 returned error can't find the container with id 3cc601da5a8778a3701b8ed917a79540e0a11cf04ac05078ae4c517f65a115b6
Nov 02 18:16:49.103477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:49.103448    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"]
Nov 02 18:16:49.342937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:49.342897    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2" event={"ID":"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af","Type":"ContainerStarted","Data":"3cc601da5a8778a3701b8ed917a79540e0a11cf04ac05078ae4c517f65a115b6"}
Nov 02 18:16:50.355494 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:50.355450    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2" event={"ID":"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af","Type":"ContainerStarted","Data":"b06108e9e184f65301022ffc17346dcd9821974c30243ad845bda08ff66e8e59"}
Nov 02 18:16:50.355961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:50.355528    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:16:50.405863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:50.378579    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2" podStartSLOduration=2.734479854 podStartE2EDuration="3.378559437s" totalImagesPullingTime="644.079583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:50.376965667 +0000 UTC m=+2622.956375054" watchObservedRunningTime="2026-11-02 18:16:50.378559437 +0000 UTC m=+2622.957968825"
Nov 02 18:16:56.371038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:16:56.371007    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:17:18.541607 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:17:18.541515    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:17:30.685969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:17:30.685936    5042 scope.go:122] "RemoveContainer" containerID="6fc3978173001747aa4cf10ff06c46771289a003db5eb40cb172967719b69ac0"
Nov 02 18:17:30.700735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:17:30.700712    5042 scope.go:122] "RemoveContainer" containerID="45d91bc35f0c5ccf19dc7ce229ea0e71af46957f493dec8e7f9fc9e7d14883e2"
Nov 02 18:17:30.713913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:17:30.713892    5042 scope.go:122] "RemoveContainer" containerID="ff9a095f8345a46a2b655d2d57b75c3785edf6cd8c8c9d5ed0037d9d7354fc96"
Nov 02 18:17:48.540720 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:17:48.540683    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:17:55.030252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:17:55.030215    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"]
Nov 02 18:17:55.035355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:17:55.035323    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ddbf560-zzmrk"]
Nov 02 18:17:55.548401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:17:55.548366    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee644284-f14a-4385-bb1e-4173b49c904a" path="/var/lib/kubelet/pods/ee644284-f14a-4385-bb1e-4173b49c904a/volumes"
Nov 02 18:18:09.641706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:09.641676    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:18:09.642344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:09.641676    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:18:09.646793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:09.646749    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:18:09.646793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:09.646762    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:18:18.577144 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:18:18.577105    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:18:47.288226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:47.288166    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:18:47.689405 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:47.487966    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:18:49.008094 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:18:49.008051    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:18:49.124665 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:49.124625    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"]
Nov 02 18:18:49.142942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:49.142909    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66af28c3-7g5r5"]
Nov 02 18:18:49.549160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:49.549126    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81af6bbf-e007-41bb-8e97-0ae620322b05" path="/var/lib/kubelet/pods/81af6bbf-e007-41bb-8e97-0ae620322b05/volumes"
Nov 02 18:18:53.168782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:53.168738    5042 generic.go:356] "Generic (PLEG): container finished" podID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerID="9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4" exitCode=0
Nov 02 18:18:53.169398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:53.168800    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" event={"ID":"19ba2ee9-18e8-4712-ad8b-da7052994a39","Type":"ContainerDied","Data":"9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"}
Nov 02 18:18:53.171263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:53.169408    5042 scope.go:122] "RemoveContainer" containerID="9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"
Nov 02 18:18:53.187880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:53.177525    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:18:53.187880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:53.177596    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:18:53.187880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:53.177628    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:18:54.184886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:54.182490    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" event={"ID":"19ba2ee9-18e8-4712-ad8b-da7052994a39","Type":"ContainerStarted","Data":"497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f"}
Nov 02 18:18:54.555785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:54.555742    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:18:54.556255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:54.556073    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Nov 02 18:18:54.556255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:54.556120    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Nov 02 18:18:55.556663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:55.556620    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Nov 02 18:18:55.557687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:55.556691    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Nov 02 18:18:56.555498 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:56.555468    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Nov 02 18:18:56.555689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:56.555519    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Nov 02 18:18:57.584865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:57.580584    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Nov 02 18:18:57.584865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:57.580658    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Nov 02 18:18:58.555238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:58.555202    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Nov 02 18:18:58.555414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:58.555261    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Nov 02 18:18:59.555114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:59.555072    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body=
Nov 02 18:18:59.555506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:18:59.555141    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused"
Nov 02 18:19:00.038186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.037024    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"]
Nov 02 18:19:00.047003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.046966    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf89ab6-bsbnj"]
Nov 02 18:19:00.523777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.523722    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"]
Nov 02 18:19:00.531031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.530998    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.534742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.534708    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:19:00.537503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.537480    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:19:00.551764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.551735    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"]
Nov 02 18:19:00.582051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.582012    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.582464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.582373    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h8dc\" (UniqueName: \"kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.582569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.582552    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.644039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.643014    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:19:00.645372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.645350    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:19:00.646519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.646490    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:19:00.684317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.684272    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h8dc\" (UniqueName: \"kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.684506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.684477    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.685162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.684589    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.685255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.685221    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.698515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.698451    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.698924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.698899    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h8dc\" (UniqueName: \"kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc\") pod \"maas-api-revoke-sub-94153825-8q94g\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:00.875485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:00.875395    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:01.457515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:01.457484    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"]
Nov 02 18:19:01.547039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:01.547006    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add1a193-ecbe-4e96-9753-f32370513e46" path="/var/lib/kubelet/pods/add1a193-ecbe-4e96-9753-f32370513e46/volumes"
Nov 02 18:19:01.547444 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:01.547418    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g" event={"ID":"790cc34c-dcde-4ac9-af55-78213fad4065","Type":"ContainerStarted","Data":"ffa3ec904a96094b80296f81b69c325a5c882df95a543ae5a06b432c5b18bb96"}
Nov 02 18:19:02.557588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:02.557555    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g" event={"ID":"790cc34c-dcde-4ac9-af55-78213fad4065","Type":"ContainerStarted","Data":"d826ef2f37d3f207c30cf119bff28114e83691044fd0906d5c3cdd836b8e4dd9"}
Nov 02 18:19:02.582118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:02.582054    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g" podStartSLOduration=2.582039667 podStartE2EDuration="2.582039667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:02.574992965 +0000 UTC m=+2755.154402418" watchObservedRunningTime="2026-11-02 18:19:02.582039667 +0000 UTC m=+2755.161449053"
Nov 02 18:19:03.568645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:03.568608    5042 generic.go:356] "Generic (PLEG): container finished" podID="790cc34c-dcde-4ac9-af55-78213fad4065" containerID="d826ef2f37d3f207c30cf119bff28114e83691044fd0906d5c3cdd836b8e4dd9" exitCode=0
Nov 02 18:19:03.569824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:03.568656    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g" event={"ID":"790cc34c-dcde-4ac9-af55-78213fad4065","Type":"ContainerDied","Data":"d826ef2f37d3f207c30cf119bff28114e83691044fd0906d5c3cdd836b8e4dd9"}
Nov 02 18:19:04.847392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.846415    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:04.882980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.882870    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc\" (UniqueName: \"kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc\") pod \"790cc34c-dcde-4ac9-af55-78213fad4065\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") "
Nov 02 18:19:04.884645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.883252    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca\") pod \"790cc34c-dcde-4ac9-af55-78213fad4065\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") "
Nov 02 18:19:04.884645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.883393    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca\") pod \"790cc34c-dcde-4ac9-af55-78213fad4065\" (UID: \"790cc34c-dcde-4ac9-af55-78213fad4065\") "
Nov 02 18:19:04.887065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.887027    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca" pod "790cc34c-dcde-4ac9-af55-78213fad4065" (UID: "790cc34c-dcde-4ac9-af55-78213fad4065"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:19:04.909754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.903674    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca" pod "790cc34c-dcde-4ac9-af55-78213fad4065" (UID: "790cc34c-dcde-4ac9-af55-78213fad4065"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:04.909754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.907594    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc" pod "790cc34c-dcde-4ac9-af55-78213fad4065" (UID: "790cc34c-dcde-4ac9-af55-78213fad4065"). InnerVolumeSpecName "kube-api-access-7h8dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:04.987972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.987920    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h8dc\" (UniqueName: \"kubernetes.io/projected/790cc34c-dcde-4ac9-af55-78213fad4065-kube-api-access-7h8dc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:19:04.988182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.987989    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:19:04.988182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:04.988009    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790cc34c-dcde-4ac9-af55-78213fad4065-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:19:05.593798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:05.593767    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"
Nov 02 18:19:05.594008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:05.593800    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g" event={"ID":"790cc34c-dcde-4ac9-af55-78213fad4065","Type":"ContainerDied","Data":"ffa3ec904a96094b80296f81b69c325a5c882df95a543ae5a06b432c5b18bb96"}
Nov 02 18:19:05.594008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:05.593872    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa3ec904a96094b80296f81b69c325a5c882df95a543ae5a06b432c5b18bb96"
Nov 02 18:19:11.027021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.026983    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"]
Nov 02 18:19:11.027937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.027899    5042 cpu_manager.go:414] "containerMap: removing container" podUID="790cc34c-dcde-4ac9-af55-78213fad4065" containerName="revoke-keys"
Nov 02 18:19:11.027937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.027920    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790cc34c-dcde-4ac9-af55-78213fad4065" containerName="revoke-keys"
Nov 02 18:19:11.028577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.028061    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="790cc34c-dcde-4ac9-af55-78213fad4065" containerName="revoke-keys"
Nov 02 18:19:11.032260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.032239    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.036982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.036952    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:19:11.037971 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.037591    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:19:11.040974 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.040952    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"]
Nov 02 18:19:11.070173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.070137    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcbx\" (UniqueName: \"kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.070881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.070224    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.070881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.070396    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.172729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.172690    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcbx\" (UniqueName: \"kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.172964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.172790    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.172964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.172883    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.173587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.173564    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.175998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.175977    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.181311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.181283    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcbx\" (UniqueName: \"kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx\") pod \"maas-api-revoke-sub-c2b01f28-rztbh\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.354644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.354560    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:11.676832 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.676793    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"]
Nov 02 18:19:11.727789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:11.727760    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh" event={"ID":"78ebd3de-013a-4b56-874d-368f71ee9149","Type":"ContainerStarted","Data":"7a6ddf8bd48f2f86e62086965afc848c31197784624f52d077445604457c9633"}
Nov 02 18:19:12.738656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:12.738611    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh" event={"ID":"78ebd3de-013a-4b56-874d-368f71ee9149","Type":"ContainerStarted","Data":"e1aa315b2a63d0f31147e85a33e4281b286717436676a9e88be01d7ddba121e0"}
Nov 02 18:19:12.756414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:12.756358    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh" podStartSLOduration=1.7563432460000001 podStartE2EDuration="1.756343246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:12.75446939 +0000 UTC m=+2765.333878777" watchObservedRunningTime="2026-11-02 18:19:12.756343246 +0000 UTC m=+2765.335752678"
Nov 02 18:19:13.748540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:13.748501    5042 generic.go:356] "Generic (PLEG): container finished" podID="78ebd3de-013a-4b56-874d-368f71ee9149" containerID="e1aa315b2a63d0f31147e85a33e4281b286717436676a9e88be01d7ddba121e0" exitCode=0
Nov 02 18:19:13.749401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:13.748580    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh" event={"ID":"78ebd3de-013a-4b56-874d-368f71ee9149","Type":"ContainerDied","Data":"e1aa315b2a63d0f31147e85a33e4281b286717436676a9e88be01d7ddba121e0"}
Nov 02 18:19:14.969257 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:14.969222    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:15.232263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.232226    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx\" (UniqueName: \"kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx\") pod \"78ebd3de-013a-4b56-874d-368f71ee9149\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") "
Nov 02 18:19:15.232460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.232311    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca\") pod \"78ebd3de-013a-4b56-874d-368f71ee9149\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") "
Nov 02 18:19:15.232460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.232348    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca\") pod \"78ebd3de-013a-4b56-874d-368f71ee9149\" (UID: \"78ebd3de-013a-4b56-874d-368f71ee9149\") "
Nov 02 18:19:15.232820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.232794    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca" pod "78ebd3de-013a-4b56-874d-368f71ee9149" (UID: "78ebd3de-013a-4b56-874d-368f71ee9149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:19:15.233316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.233285    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:19:15.234911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.234888    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca" pod "78ebd3de-013a-4b56-874d-368f71ee9149" (UID: "78ebd3de-013a-4b56-874d-368f71ee9149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:15.234996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.234919    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx" pod "78ebd3de-013a-4b56-874d-368f71ee9149" (UID: "78ebd3de-013a-4b56-874d-368f71ee9149"). InnerVolumeSpecName "kube-api-access-tpcbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:15.335399 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.335355    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcbx\" (UniqueName: \"kubernetes.io/projected/78ebd3de-013a-4b56-874d-368f71ee9149-kube-api-access-tpcbx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:19:15.335399 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.335385    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ebd3de-013a-4b56-874d-368f71ee9149-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:19:15.770269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.770222    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh" event={"ID":"78ebd3de-013a-4b56-874d-368f71ee9149","Type":"ContainerDied","Data":"7a6ddf8bd48f2f86e62086965afc848c31197784624f52d077445604457c9633"}
Nov 02 18:19:15.770459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.770277    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"
Nov 02 18:19:15.770459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:15.770311    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6ddf8bd48f2f86e62086965afc848c31197784624f52d077445604457c9633"
Nov 02 18:19:18.555823 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:19:18.555764    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:19:30.812865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:30.807911    5042 scope.go:122] "RemoveContainer" containerID="537e480912df2a38b24af8d06eadffb9a31cea8d25e33f732b921784a8b986bf"
Nov 02 18:19:37.871057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:37.870981    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:19:37.920488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:37.917957    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:19:38.038686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:38.038639    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:19:38.075994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:38.075945    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:19:38.148326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:38.148233    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:19:40.028886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:40.028827    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"]
Nov 02 18:19:40.034068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:40.034036    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85b8b498-b5xz2"]
Nov 02 18:19:41.549700 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:41.549662    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0610f7f6-b2f5-4c32-ac73-1a79a5d036f3" path="/var/lib/kubelet/pods/0610f7f6-b2f5-4c32-ac73-1a79a5d036f3/volumes"
Nov 02 18:19:48.557500 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:19:48.557455    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:19:49.027531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:49.027493    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"]
Nov 02 18:19:49.032609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:49.032575    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-015bf5e2-49fgk"]
Nov 02 18:19:49.546013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:19:49.545979    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3078b6e0-f97d-4fb0-88c9-6d8f07b048a3" path="/var/lib/kubelet/pods/3078b6e0-f97d-4fb0-88c9-6d8f07b048a3/volumes"
Nov 02 18:20:07.689907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.687237    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"]
Nov 02 18:20:07.689907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.688599    5042 cpu_manager.go:414] "containerMap: removing container" podUID="78ebd3de-013a-4b56-874d-368f71ee9149" containerName="revoke-keys"
Nov 02 18:20:07.689907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.688620    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ebd3de-013a-4b56-874d-368f71ee9149" containerName="revoke-keys"
Nov 02 18:20:07.689907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.688833    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ebd3de-013a-4b56-874d-368f71ee9149" containerName="revoke-keys"
Nov 02 18:20:07.695047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.695021    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.700197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.700165    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"]
Nov 02 18:20:07.701888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.700406    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:20:07.701888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.700441    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:20:07.765266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.765215    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.765482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.765394    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qj7\" (UniqueName: \"kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.766194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.765480    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.866600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.866566    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qj7\" (UniqueName: \"kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.867365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.866658    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.867365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.866901    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.871072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.871046    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.878125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.878096    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qj7\" (UniqueName: \"kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:07.878944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:07.878396    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d50fc2-ln9pf\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:08.023244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:08.023158    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:08.343664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:08.343631    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"]
Nov 02 18:20:08.345396 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:20:08.345361    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709756ae_aedd_4054_a282_c010a187b4d3.slice/crio-5993e041be8f074a6a8ba1dc13d2f581cd1a0ecabd092f47f6c1092dea9821de WatchSource:0}: Error finding container 5993e041be8f074a6a8ba1dc13d2f581cd1a0ecabd092f47f6c1092dea9821de: Status 404 returned error can't find the container with id 5993e041be8f074a6a8ba1dc13d2f581cd1a0ecabd092f47f6c1092dea9821de
Nov 02 18:20:09.099949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:09.099914    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf" event={"ID":"709756ae-aedd-4054-a282-c010a187b4d3","Type":"ContainerStarted","Data":"79a41f3f27f4fbb0e33bcbd9e20258e10f39d8b56b63ea766cca7338db350810"}
Nov 02 18:20:09.101938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:09.099970    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf" event={"ID":"709756ae-aedd-4054-a282-c010a187b4d3","Type":"ContainerStarted","Data":"5993e041be8f074a6a8ba1dc13d2f581cd1a0ecabd092f47f6c1092dea9821de"}
Nov 02 18:20:09.120211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:09.120155    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf" podStartSLOduration=2.120140558 podStartE2EDuration="2.120140558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:09.118216478 +0000 UTC m=+2821.697625861" watchObservedRunningTime="2026-11-02 18:20:09.120140558 +0000 UTC m=+2821.699549945"
Nov 02 18:20:10.184663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:10.184622    5042 generic.go:356] "Generic (PLEG): container finished" podID="709756ae-aedd-4054-a282-c010a187b4d3" containerID="79a41f3f27f4fbb0e33bcbd9e20258e10f39d8b56b63ea766cca7338db350810" exitCode=0
Nov 02 18:20:10.185815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:10.185656    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf" event={"ID":"709756ae-aedd-4054-a282-c010a187b4d3","Type":"ContainerDied","Data":"79a41f3f27f4fbb0e33bcbd9e20258e10f39d8b56b63ea766cca7338db350810"}
Nov 02 18:20:11.415318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.415285    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:11.503573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.503463    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca\") pod \"709756ae-aedd-4054-a282-c010a187b4d3\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") "
Nov 02 18:20:11.503830 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.503758    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7\" (UniqueName: \"kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7\") pod \"709756ae-aedd-4054-a282-c010a187b4d3\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") "
Nov 02 18:20:11.503830 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.503801    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca\") pod \"709756ae-aedd-4054-a282-c010a187b4d3\" (UID: \"709756ae-aedd-4054-a282-c010a187b4d3\") "
Nov 02 18:20:11.504475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.504446    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca" pod "709756ae-aedd-4054-a282-c010a187b4d3" (UID: "709756ae-aedd-4054-a282-c010a187b4d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:11.505089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.505015    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:11.510893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.510295    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca" pod "709756ae-aedd-4054-a282-c010a187b4d3" (UID: "709756ae-aedd-4054-a282-c010a187b4d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:11.511538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.511501    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7" pod "709756ae-aedd-4054-a282-c010a187b4d3" (UID: "709756ae-aedd-4054-a282-c010a187b4d3"). InnerVolumeSpecName "kube-api-access-t2qj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:11.645382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.627310    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qj7\" (UniqueName: \"kubernetes.io/projected/709756ae-aedd-4054-a282-c010a187b4d3-kube-api-access-t2qj7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:11.645382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:11.627345    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/709756ae-aedd-4054-a282-c010a187b4d3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:12.209493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:12.209462    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"
Nov 02 18:20:12.209493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:12.209467    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf" event={"ID":"709756ae-aedd-4054-a282-c010a187b4d3","Type":"ContainerDied","Data":"5993e041be8f074a6a8ba1dc13d2f581cd1a0ecabd092f47f6c1092dea9821de"}
Nov 02 18:20:12.210359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:12.209535    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5993e041be8f074a6a8ba1dc13d2f581cd1a0ecabd092f47f6c1092dea9821de"
Nov 02 18:20:18.551473 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:20:18.551381    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:20:19.335105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.335051    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"]
Nov 02 18:20:19.336172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.336142    5042 cpu_manager.go:414] "containerMap: removing container" podUID="709756ae-aedd-4054-a282-c010a187b4d3" containerName="revoke-keys"
Nov 02 18:20:19.336172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.336171    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709756ae-aedd-4054-a282-c010a187b4d3" containerName="revoke-keys"
Nov 02 18:20:19.336828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.336427    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="709756ae-aedd-4054-a282-c010a187b4d3" containerName="revoke-keys"
Nov 02 18:20:19.340747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.340722    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.364874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.347075    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"]
Nov 02 18:20:19.364874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.347399    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:20:19.364874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.348054    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:20:19.410265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.410206    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.410508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.410431    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsf4z\" (UniqueName: \"kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.410508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.410460    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.512511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.512472    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsf4z\" (UniqueName: \"kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.512751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.512523    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.512751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.512704    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.513930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.513903    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.516945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.516918    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.521365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.521340    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsf4z\" (UniqueName: \"kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z\") pod \"maas-api-revoke-sub-f19ad422-5vvgj\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:19.700491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:19.700444    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:20.127284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:20.127254    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"]
Nov 02 18:20:20.127882 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:20:20.127831    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c24635b_b6fd_49d2_a23b_ef78a00b20cd.slice/crio-d56d5d26377b643f9f4e1b98428335f47b07c7b6fdfe8aa60d00cf6c99704f6d WatchSource:0}: Error finding container d56d5d26377b643f9f4e1b98428335f47b07c7b6fdfe8aa60d00cf6c99704f6d: Status 404 returned error can't find the container with id d56d5d26377b643f9f4e1b98428335f47b07c7b6fdfe8aa60d00cf6c99704f6d
Nov 02 18:20:20.290283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:20.290243    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj" event={"ID":"1c24635b-b6fd-49d2-a23b-ef78a00b20cd","Type":"ContainerStarted","Data":"d56d5d26377b643f9f4e1b98428335f47b07c7b6fdfe8aa60d00cf6c99704f6d"}
Nov 02 18:20:21.324869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:21.323696    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj" event={"ID":"1c24635b-b6fd-49d2-a23b-ef78a00b20cd","Type":"ContainerStarted","Data":"1f5120357436a1b4d0c6ccc9196765b4e1d017c33f12d757391d52c373f1441e"}
Nov 02 18:20:21.364089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:21.364020    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj" podStartSLOduration=2.363997274 podStartE2EDuration="2.363997274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:21.348258171 +0000 UTC m=+2833.927667559" watchObservedRunningTime="2026-11-02 18:20:21.363997274 +0000 UTC m=+2833.943406661"
Nov 02 18:20:22.334239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:22.334203    5042 generic.go:356] "Generic (PLEG): container finished" podID="1c24635b-b6fd-49d2-a23b-ef78a00b20cd" containerID="1f5120357436a1b4d0c6ccc9196765b4e1d017c33f12d757391d52c373f1441e" exitCode=0
Nov 02 18:20:22.334780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:22.334304    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj" event={"ID":"1c24635b-b6fd-49d2-a23b-ef78a00b20cd","Type":"ContainerDied","Data":"1f5120357436a1b4d0c6ccc9196765b4e1d017c33f12d757391d52c373f1441e"}
Nov 02 18:20:23.569059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.569031    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:23.721123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.721083    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca\") pod \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") "
Nov 02 18:20:23.721376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.721315    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z\" (UniqueName: \"kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z\") pod \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") "
Nov 02 18:20:23.721482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.721454    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca\") pod \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\" (UID: \"1c24635b-b6fd-49d2-a23b-ef78a00b20cd\") "
Nov 02 18:20:23.725151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.722806    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca" pod "1c24635b-b6fd-49d2-a23b-ef78a00b20cd" (UID: "1c24635b-b6fd-49d2-a23b-ef78a00b20cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:23.726143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.725711    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z" pod "1c24635b-b6fd-49d2-a23b-ef78a00b20cd" (UID: "1c24635b-b6fd-49d2-a23b-ef78a00b20cd"). InnerVolumeSpecName "kube-api-access-gsf4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:23.727685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.727474    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca" pod "1c24635b-b6fd-49d2-a23b-ef78a00b20cd" (UID: "1c24635b-b6fd-49d2-a23b-ef78a00b20cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:23.824799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.824756    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsf4z\" (UniqueName: \"kubernetes.io/projected/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-kube-api-access-gsf4z\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:23.824799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.824797    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:23.825049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:23.824812    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c24635b-b6fd-49d2-a23b-ef78a00b20cd-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:24.406918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:24.404370    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj" event={"ID":"1c24635b-b6fd-49d2-a23b-ef78a00b20cd","Type":"ContainerDied","Data":"d56d5d26377b643f9f4e1b98428335f47b07c7b6fdfe8aa60d00cf6c99704f6d"}
Nov 02 18:20:24.406918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:24.404466    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d56d5d26377b643f9f4e1b98428335f47b07c7b6fdfe8aa60d00cf6c99704f6d"
Nov 02 18:20:24.406918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:24.404623    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"
Nov 02 18:20:31.132688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:31.132641    5042 scope.go:122] "RemoveContainer" containerID="befd5149bd532c63bbe001d78c1e0fd8885e8258091b5ebef639fdfa2e50ab66"
Nov 02 18:20:31.156477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:31.156442    5042 scope.go:122] "RemoveContainer" containerID="1f964f3001cf3f1fc49d5d8c29d44f30b5cdca8c44caf2f3f14369c250a08646"
Nov 02 18:20:45.032436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:45.032400    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"]
Nov 02 18:20:45.038009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:45.037982    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-608e0df0-v25bq"]
Nov 02 18:20:45.547693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:45.547657    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da41bf0f-fc8c-4703-8c9b-b434990f054d" path="/var/lib/kubelet/pods/da41bf0f-fc8c-4703-8c9b-b434990f054d/volumes"
Nov 02 18:20:48.666091 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:20:48.666056    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:20:49.856864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:49.856812    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"]
Nov 02 18:20:49.869070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:49.869025    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" podUID="5a60e20b-67cf-435b-8385-5df6b10ab242" containerName="maas-api" containerID="cri-o://efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02" gracePeriod=30
Nov 02 18:20:50.865738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.865705    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:20:50.872221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.872183    5042 generic.go:356] "Generic (PLEG): container finished" podID="5a60e20b-67cf-435b-8385-5df6b10ab242" containerID="efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02" exitCode=0
Nov 02 18:20:50.872415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.872224    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"
Nov 02 18:20:50.872415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.872276    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" event={"ID":"5a60e20b-67cf-435b-8385-5df6b10ab242","Type":"ContainerDied","Data":"efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02"}
Nov 02 18:20:50.872415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.872336    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" event={"ID":"5a60e20b-67cf-435b-8385-5df6b10ab242","Type":"ContainerDied","Data":"e36e058ac1fd17908aea883eee004862449e00ae8421fa6c5970317b16e087be"}
Nov 02 18:20:50.872415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.872368    5042 scope.go:122] "RemoveContainer" containerID="efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02"
Nov 02 18:20:50.902639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.902614    5042 scope.go:122] "RemoveContainer" containerID="efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02"
Nov 02 18:20:50.904930 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:20:50.904894    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02\": container with ID starting with efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02 not found: ID does not exist" containerID="efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02"
Nov 02 18:20:50.905070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.904940    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02"} err="failed to get container status \"efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02\": rpc error: code = NotFound desc = could not find container \"efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02\": container with ID starting with efffc5ce5951bc81ffe5723c33ab5b18eaced9df5f7c2a48935b274dfa369d02 not found: ID does not exist"
Nov 02 18:20:50.911712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.911677    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"]
Nov 02 18:20:50.913734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.912950    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5a60e20b-67cf-435b-8385-5df6b10ab242" containerName="maas-api"
Nov 02 18:20:50.913734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.912974    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a60e20b-67cf-435b-8385-5df6b10ab242" containerName="maas-api"
Nov 02 18:20:50.913734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.913032    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1c24635b-b6fd-49d2-a23b-ef78a00b20cd" containerName="revoke-keys"
Nov 02 18:20:50.913734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.913040    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c24635b-b6fd-49d2-a23b-ef78a00b20cd" containerName="revoke-keys"
Nov 02 18:20:50.913734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.913278    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a60e20b-67cf-435b-8385-5df6b10ab242" containerName="maas-api"
Nov 02 18:20:50.913734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.913301    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c24635b-b6fd-49d2-a23b-ef78a00b20cd" containerName="revoke-keys"
Nov 02 18:20:50.917036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.916081    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca\") pod \"5a60e20b-67cf-435b-8385-5df6b10ab242\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") "
Nov 02 18:20:50.917036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.916407    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs\") pod \"5a60e20b-67cf-435b-8385-5df6b10ab242\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") "
Nov 02 18:20:50.917036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.916490    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls\") pod \"5a60e20b-67cf-435b-8385-5df6b10ab242\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") "
Nov 02 18:20:50.917036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.916545    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4\" (UniqueName: \"kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4\") pod \"5a60e20b-67cf-435b-8385-5df6b10ab242\" (UID: \"5a60e20b-67cf-435b-8385-5df6b10ab242\") "
Nov 02 18:20:50.927983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.921951    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca" pod "5a60e20b-67cf-435b-8385-5df6b10ab242" (UID: "5a60e20b-67cf-435b-8385-5df6b10ab242"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:50.927983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.922130    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"]
Nov 02 18:20:50.927983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.922247    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:50.927983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.924680    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4" pod "5a60e20b-67cf-435b-8385-5df6b10ab242" (UID: "5a60e20b-67cf-435b-8385-5df6b10ab242"). InnerVolumeSpecName "kube-api-access-fvhg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:50.927983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.924741    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs" pod "5a60e20b-67cf-435b-8385-5df6b10ab242" (UID: "5a60e20b-67cf-435b-8385-5df6b10ab242"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:50.927983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.927324    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-312f21-openshift-default-dockercfg-plxfc\""
Nov 02 18:20:50.929063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:50.928797    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls" pod "5a60e20b-67cf-435b-8385-5df6b10ab242" (UID: "5a60e20b-67cf-435b-8385-5df6b10ab242"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:51.019247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019199    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.019459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019347    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.019459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019382    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.019459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019409    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slkbm\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.019652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019567    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.019652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019599    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019690    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019732    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019818    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019940    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019952    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a60e20b-67cf-435b-8385-5df6b10ab242-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019962    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvhg4\" (UniqueName: \"kubernetes.io/projected/5a60e20b-67cf-435b-8385-5df6b10ab242-kube-api-access-fvhg4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:51.020946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.019973    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a60e20b-67cf-435b-8385-5df6b10ab242-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:20:51.161925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.150574    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.216576    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.216701    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.216786    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.217029    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.217136    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.217276    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.217318    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.217356    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slkbm\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.236443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.218716    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.251207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.250827    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.251207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.251162    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.251661 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.251616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.278258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.278222    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.290917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.283374    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.290917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.290770    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.317273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.310775    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.318146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.318116    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slkbm\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm\") pod \"e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") " pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.463986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.463916    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"]
Nov 02 18:20:51.480825 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.480585    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp"]
Nov 02 18:20:51.583652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.583609    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:51.866020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:51.865932    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a60e20b-67cf-435b-8385-5df6b10ab242" path="/var/lib/kubelet/pods/5a60e20b-67cf-435b-8385-5df6b10ab242/volumes"
Nov 02 18:20:52.326494 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:20:52.326453    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d3f0cc_9bff_48aa_a233_c572e9097b13.slice/crio-6e3ebb7b63f720187e60cf5e6a0862073063647b0beee996ebc1c89cbc7c3557 WatchSource:0}: Error finding container 6e3ebb7b63f720187e60cf5e6a0862073063647b0beee996ebc1c89cbc7c3557: Status 404 returned error can't find the container with id 6e3ebb7b63f720187e60cf5e6a0862073063647b0beee996ebc1c89cbc7c3557
Nov 02 18:20:52.329594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:52.329566    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"]
Nov 02 18:20:52.330803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:52.330764    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:20:52.330918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:52.330900    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:20:52.330971 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:52.330955    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:20:53.108595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.108546    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" event={"ID":"f1d3f0cc-9bff-48aa-a233-c572e9097b13","Type":"ContainerStarted","Data":"1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220"}
Nov 02 18:20:53.109137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.108622    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" event={"ID":"f1d3f0cc-9bff-48aa-a233-c572e9097b13","Type":"ContainerStarted","Data":"6e3ebb7b63f720187e60cf5e6a0862073063647b0beee996ebc1c89cbc7c3557"}
Nov 02 18:20:53.137174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.137113    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podStartSLOduration=150.137096181 podStartE2EDuration="2m30.137096181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:53.133289017 +0000 UTC m=+2865.712698402" watchObservedRunningTime="2026-11-02 18:20:53.137096181 +0000 UTC m=+2865.716505567"
Nov 02 18:20:53.592151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.592115    5042 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 18:20:53.593986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.593932    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 18:20:53.618732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.618699    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:53.741868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:53.741812    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-78383f-76b79875-5ggvp" podUID="5a60e20b-67cf-435b-8385-5df6b10ab242" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.29:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:20:54.585951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:54.585432    5042 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 18:20:54.585951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:54.585496    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 18:20:55.030298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:55.030259    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"]
Nov 02 18:20:55.036236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:55.036207    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44364b1c-pcv9v"]
Nov 02 18:20:55.550042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:55.550001    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54a985f-9278-4a71-a016-807d2ecf6e19" path="/var/lib/kubelet/pods/c54a985f-9278-4a71-a016-807d2ecf6e19/volumes"
Nov 02 18:20:55.585866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:55.585271    5042 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 18:20:55.585866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:55.585339    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 18:20:56.588927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:56.585439    5042 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 18:20:56.588927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:56.585503    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 18:20:57.598869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:57.598056    5042 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 18:20:57.598869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:57.598117    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 18:20:58.587350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:58.587313    5042 patch_prober.go:29] interesting pod/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused" start-of-body=
Nov 02 18:20:58.587642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:58.587606    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.62:15021/healthz/ready\": dial tcp 10.132.1.62:15021: connect: connection refused"
Nov 02 18:20:59.591401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:59.591365    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:59.591989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:59.591970    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:20:59.592892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:20:59.592863    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:21:16.567781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.567738    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"]
Nov 02 18:21:16.573532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.573503    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.580132 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.580095    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:21:16.580321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.580220    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:21:16.582746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.582712    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"]
Nov 02 18:21:16.735121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.735080    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25dx\" (UniqueName: \"kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.735316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.735260    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.735374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.735315    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.844601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.844498    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25dx\" (UniqueName: \"kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.859601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.859566    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.860764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.860737    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.861765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.860603    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.901528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.901490    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:16.924269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:16.924233    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25dx\" (UniqueName: \"kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx\") pod \"maas-api-revoke-sub-2426eb15-cbjkm\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:17.205989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:17.205922    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:17.503526 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:21:17.503493    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f96ead_b24e_490c_8420_e7a04d030652.slice/crio-eb31822d020ac70d485196ef7600d02e487b9fc6b936b2ec7608e47cf5bc8114 WatchSource:0}: Error finding container eb31822d020ac70d485196ef7600d02e487b9fc6b936b2ec7608e47cf5bc8114: Status 404 returned error can't find the container with id eb31822d020ac70d485196ef7600d02e487b9fc6b936b2ec7608e47cf5bc8114
Nov 02 18:21:17.504899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:17.504864    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"]
Nov 02 18:21:17.721406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:17.721370    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm" event={"ID":"01f96ead-b24e-490c-8420-e7a04d030652","Type":"ContainerStarted","Data":"eb31822d020ac70d485196ef7600d02e487b9fc6b936b2ec7608e47cf5bc8114"}
Nov 02 18:21:18.541094 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:18.541055    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:21:18.743590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:18.743546    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm" event={"ID":"01f96ead-b24e-490c-8420-e7a04d030652","Type":"ContainerStarted","Data":"defe0c92aa6569adb30c0ab6346789fe124ac7903b0d01e1d557fc56ac747281"}
Nov 02 18:21:18.765442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:18.765380    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm" podStartSLOduration=2.7653646309999997 podStartE2EDuration="2.765364631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:18.761142903 +0000 UTC m=+2891.340552292" watchObservedRunningTime="2026-11-02 18:21:18.765364631 +0000 UTC m=+2891.344774017"
Nov 02 18:21:19.797040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:19.796913    5042 generic.go:356] "Generic (PLEG): container finished" podID="01f96ead-b24e-490c-8420-e7a04d030652" containerID="defe0c92aa6569adb30c0ab6346789fe124ac7903b0d01e1d557fc56ac747281" exitCode=0
Nov 02 18:21:19.797583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:19.797057    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm" event={"ID":"01f96ead-b24e-490c-8420-e7a04d030652","Type":"ContainerDied","Data":"defe0c92aa6569adb30c0ab6346789fe124ac7903b0d01e1d557fc56ac747281"}
Nov 02 18:21:21.063011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.062922    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:21.232866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.230631    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx\" (UniqueName: \"kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx\") pod \"01f96ead-b24e-490c-8420-e7a04d030652\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") "
Nov 02 18:21:21.232866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.230754    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca\") pod \"01f96ead-b24e-490c-8420-e7a04d030652\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") "
Nov 02 18:21:21.232866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.231160    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca\") pod \"01f96ead-b24e-490c-8420-e7a04d030652\" (UID: \"01f96ead-b24e-490c-8420-e7a04d030652\") "
Nov 02 18:21:21.274865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.257264    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca" pod "01f96ead-b24e-490c-8420-e7a04d030652" (UID: "01f96ead-b24e-490c-8420-e7a04d030652"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:21.303862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.288805    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:21.644622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.644535    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx" pod "01f96ead-b24e-490c-8420-e7a04d030652" (UID: "01f96ead-b24e-490c-8420-e7a04d030652"). InnerVolumeSpecName "kube-api-access-p25dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:21.645711 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.645681    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca" pod "01f96ead-b24e-490c-8420-e7a04d030652" (UID: "01f96ead-b24e-490c-8420-e7a04d030652"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:21.810034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.809381    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25dx\" (UniqueName: \"kubernetes.io/projected/01f96ead-b24e-490c-8420-e7a04d030652-kube-api-access-p25dx\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:21.810034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.809432    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01f96ead-b24e-490c-8420-e7a04d030652-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:21.838532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.838500    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm" event={"ID":"01f96ead-b24e-490c-8420-e7a04d030652","Type":"ContainerDied","Data":"eb31822d020ac70d485196ef7600d02e487b9fc6b936b2ec7608e47cf5bc8114"}
Nov 02 18:21:21.838753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.838740    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb31822d020ac70d485196ef7600d02e487b9fc6b936b2ec7608e47cf5bc8114"
Nov 02 18:21:21.838931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:21.838917    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"
Nov 02 18:21:25.435934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:25.435889    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"]
Nov 02 18:21:25.436531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:25.436169    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy" containerID="cri-o://f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895" gracePeriod=30
Nov 02 18:21:25.455183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:25.455146    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"]
Nov 02 18:21:25.456114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:25.455574    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" containerID="cri-o://4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58" gracePeriod=30
Nov 02 18:21:26.753396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:26.753362    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"]
Nov 02 18:21:26.754642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:26.753669    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy" containerID="cri-o://6873572cd2f7d3b465cc055bc615a163e703bad331990bfd253a0cf43e5a0f9b" gracePeriod=30
Nov 02 18:21:26.759349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:26.759311    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"]
Nov 02 18:21:26.759603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:26.759574    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy" containerID="cri-o://a9d28fa02ee73fbae4993a6b58af4f3851ad467579cc8f0d1dbe422c4eb987d3" gracePeriod=30
Nov 02 18:21:27.683226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.683190    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"]
Nov 02 18:21:27.684236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.684211    5042 cpu_manager.go:414] "containerMap: removing container" podUID="01f96ead-b24e-490c-8420-e7a04d030652" containerName="revoke-keys"
Nov 02 18:21:27.684236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.684235    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f96ead-b24e-490c-8420-e7a04d030652" containerName="revoke-keys"
Nov 02 18:21:27.684464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.684447    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f96ead-b24e-490c-8420-e7a04d030652" containerName="revoke-keys"
Nov 02 18:21:27.690058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.690028    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.694749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.694725    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:21:27.694950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.694779    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:21:27.698281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.698253    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"]
Nov 02 18:21:27.775079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.775043    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sg7d\" (UniqueName: \"kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.775572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.775099    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.775572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.775209    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.877238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.877205    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sg7d\" (UniqueName: \"kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.877505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.877249    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.877505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.877285    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.878510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.878068    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.880477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.880443    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:27.886482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:27.886458    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sg7d\" (UniqueName: \"kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d\") pod \"maas-api-revoke-sub-7f6f59ca-vdccm\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:28.033197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:28.033158    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:28.412977 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:21:28.412939    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e49cc75_fea8_4d97_b290_7922411562b3.slice/crio-4249203ca775de728a82acd8221d26cab3af94eb07306be03c57609a1f467b6f WatchSource:0}: Error finding container 4249203ca775de728a82acd8221d26cab3af94eb07306be03c57609a1f467b6f: Status 404 returned error can't find the container with id 4249203ca775de728a82acd8221d26cab3af94eb07306be03c57609a1f467b6f
Nov 02 18:21:28.413222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:28.413196    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"]
Nov 02 18:21:28.964394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:28.964352    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm" event={"ID":"1e49cc75-fea8-4d97-b290-7922411562b3","Type":"ContainerStarted","Data":"d41f81d11973ec26e0fb8910db50f035a0a939a53a7e4f10c1b341eadaadc681"}
Nov 02 18:21:28.964929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:28.964416    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm" event={"ID":"1e49cc75-fea8-4d97-b290-7922411562b3","Type":"ContainerStarted","Data":"4249203ca775de728a82acd8221d26cab3af94eb07306be03c57609a1f467b6f"}
Nov 02 18:21:28.981683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:28.981627    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm" podStartSLOduration=1.981614177 podStartE2EDuration="1.981614177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:28.980779722 +0000 UTC m=+2901.560189110" watchObservedRunningTime="2026-11-02 18:21:28.981614177 +0000 UTC m=+2901.561023562"
Nov 02 18:21:29.976439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:29.976398    5042 generic.go:356] "Generic (PLEG): container finished" podID="1e49cc75-fea8-4d97-b290-7922411562b3" containerID="d41f81d11973ec26e0fb8910db50f035a0a939a53a7e4f10c1b341eadaadc681" exitCode=0
Nov 02 18:21:29.977069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:29.976482    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm" event={"ID":"1e49cc75-fea8-4d97-b290-7922411562b3","Type":"ContainerDied","Data":"d41f81d11973ec26e0fb8910db50f035a0a939a53a7e4f10c1b341eadaadc681"}
Nov 02 18:21:30.197097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.197056    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:21:30.197290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.197122    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:21:30.806434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.806403    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 18:21:30.860794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.860757    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"]
Nov 02 18:21:30.862185 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.861767    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy"
Nov 02 18:21:30.862185 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.861786    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy"
Nov 02 18:21:30.862185 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.862097    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerName="istio-proxy"
Nov 02 18:21:30.870096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.870068    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 18:21:30.870372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.870353    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:30.892869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.885056    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"]
Nov 02 18:21:30.908865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.906291    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"]
Nov 02 18:21:30.908865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.907160    5042 cpu_manager.go:414] "containerMap: removing container" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy"
Nov 02 18:21:30.908865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.907177    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy"
Nov 02 18:21:30.908865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.907516    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy"
Nov 02 18:21:30.908865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.907533    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy"
Nov 02 18:21:30.937583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.933128    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:30.937583 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.935974    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"]
Nov 02 18:21:30.943227 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.943184    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2966e7-openshift-default-dockercfg-xd7qq\""
Nov 02 18:21:30.961120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.960557    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.961120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.960805    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.961760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.961713    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.962739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.961779    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.962739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.961815    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.962739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.961888    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.962739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.962077    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.962739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.962148    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.962739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.962278    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo\") pod \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\" (UID: \"ff95a5d7-5e13-4987-9a9f-6c10ced46438\") "
Nov 02 18:21:30.963221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.962769    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:30.966220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.964989    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:30.966418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.965969    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:30.966549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.965175    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:30.966738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.966702    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:30.966937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.966736    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:30.968686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.968662    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:30.969810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.969790    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "kube-api-access-zbfmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:30.971321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.971295    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:30.974916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.971800    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token" pod "ff95a5d7-5e13-4987-9a9f-6c10ced46438" (UID: "ff95a5d7-5e13-4987-9a9f-6c10ced46438"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:30.996078 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.995970    5042 generic.go:356] "Generic (PLEG): container finished" podID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" containerID="f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895" exitCode=0
Nov 02 18:21:30.996540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.996228    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"
Nov 02 18:21:30.996540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.996247    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" event={"ID":"ff95a5d7-5e13-4987-9a9f-6c10ced46438","Type":"ContainerDied","Data":"f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895"}
Nov 02 18:21:30.996540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.996301    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q" event={"ID":"ff95a5d7-5e13-4987-9a9f-6c10ced46438","Type":"ContainerDied","Data":"83a0053a34c27d1ced15a65fac61d7eb7757411e6b7eacad5a77ea43471e0106"}
Nov 02 18:21:30.996540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:30.996332    5042 scope.go:122] "RemoveContainer" containerID="f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895"
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.039306    5042 generic.go:356] "Generic (PLEG): container finished" podID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerID="4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58" exitCode=0
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.039717    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.047416    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" event={"ID":"db2d4865-b81a-4d68-9dde-1bc29f58bde7","Type":"ContainerDied","Data":"4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58"}
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.047491    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25" event={"ID":"db2d4865-b81a-4d68-9dde-1bc29f58bde7","Type":"ContainerDied","Data":"f58ba23bd43d181eb3924b8ce95b816995fe3662b44e9bfea954a9712a0a6739"}
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.084453    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.084570    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.084676    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.086206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.084785    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.146824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.145938    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:31.146824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.146273    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:31.147352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.147316    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.148030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.147933    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.148226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.148183    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.148448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.148413    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.148604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.148566    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy\") pod \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\" (UID: \"db2d4865-b81a-4d68-9dde-1bc29f58bde7\") "
Nov 02 18:21:31.149336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.148834    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:31.149955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.149886    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.150260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.150158    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.150260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.150216    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.150646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.150552    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5s6\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.150646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.150615    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.150897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.150813    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.151129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.151065    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.151286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.151228    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.151442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.151264    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.158495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.151419    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.159142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159111    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.159263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159198    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd44c\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.159263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159228    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.159453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159435    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.159705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159579    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.159705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159651    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.159920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159719    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.159920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.159777    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.160057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160034    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160058    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160074    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160088    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbfmp\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-kube-api-access-zbfmp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160105    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160120    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160133    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff95a5d7-5e13-4987-9a9f-6c10ced46438-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160148    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160163    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160178    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff95a5d7-5e13-4987-9a9f-6c10ced46438-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160191    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.160736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160677    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:31.160736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.160712    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:31.161439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.161380    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:31.163353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.163191    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:31.168890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.165110    5042 scope.go:122] "RemoveContainer" containerID="f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895"
Nov 02 18:21:31.168890 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:31.165587    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895\": container with ID starting with f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895 not found: ID does not exist" containerID="f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895"
Nov 02 18:21:31.168890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.165624    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895"} err="failed to get container status \"f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895\": rpc error: code = NotFound desc = could not find container \"f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895\": container with ID starting with f794c899a49de37d8b0ff58e54cdb6c846a68c02045c8ee922ef7b205ea6b895 not found: ID does not exist"
Nov 02 18:21:31.168890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.165653    5042 scope.go:122] "RemoveContainer" containerID="4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58"
Nov 02 18:21:31.170501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.170219    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:31.170501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.170272    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv" pod "db2d4865-b81a-4d68-9dde-1bc29f58bde7" (UID: "db2d4865-b81a-4d68-9dde-1bc29f58bde7"). InnerVolumeSpecName "kube-api-access-k4rzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:31.198111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.198040    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"]
Nov 02 18:21:31.198678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.198658    5042 scope.go:122] "RemoveContainer" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"
Nov 02 18:21:31.205791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.205688    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-pvs6q"]
Nov 02 18:21:31.214511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.214473    5042 scope.go:122] "RemoveContainer" containerID="62fc17b5ddfcd747d002f63936c203d1251e0cd647b50ee4a3306325d8fea2b7"
Nov 02 18:21:31.216432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.216409    5042 scope.go:122] "RemoveContainer" containerID="4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58"
Nov 02 18:21:31.216832 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:31.216804    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58\": container with ID starting with 4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58 not found: ID does not exist" containerID="4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58"
Nov 02 18:21:31.217013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.216876    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58"} err="failed to get container status \"4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58\": rpc error: code = NotFound desc = could not find container \"4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58\": container with ID starting with 4e62831a7b34f56717b5ff4d65e0b31c97cdee54770ffee50223676b8b95bc58 not found: ID does not exist"
Nov 02 18:21:31.217013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.216903    5042 scope.go:122] "RemoveContainer" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"
Nov 02 18:21:31.217735 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:31.217291    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470\": container with ID starting with 1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470 not found: ID does not exist" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"
Nov 02 18:21:31.217735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.217320    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"} err="failed to get container status \"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470\": rpc error: code = NotFound desc = could not find container \"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470\": container with ID starting with 1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470 not found: ID does not exist"
Nov 02 18:21:31.230793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.230764    5042 scope.go:122] "RemoveContainer" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"
Nov 02 18:21:31.231330 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:31.231302    5042 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470\": rpc error: code = NotFound desc = could not find container \"1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470\": container with ID starting with 1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470 not found: ID does not exist" containerID="1ecfdebe49d6ac6390bb214fe30116b13b81b808ef42c8f0456e0f8465044470"
Nov 02 18:21:31.231461 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.231335    5042 scope.go:122] "RemoveContainer" containerID="d15ce962118e32a9dd7b3352e365b64db88df8650d7534beb18e1db9c138af48"
Nov 02 18:21:31.338552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.337930    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.339138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.338774    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.339138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.338915    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.339277    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.339302    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.339583    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.341514    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.341597    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342007    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.341722    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342105    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.342585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342425    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342617    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5s6\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342663    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342702    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342752    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342797    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342826    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342887    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342939    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.342998    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd44c\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343027    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343058    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343320    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343339    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343353    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343370    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343387    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rzv\" (UniqueName: \"kubernetes.io/projected/db2d4865-b81a-4d68-9dde-1bc29f58bde7-kube-api-access-k4rzv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343404    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2d4865-b81a-4d68-9dde-1bc29f58bde7-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343525    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.343811    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.344499    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.352706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.346295    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.354784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.354429    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.354920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.354801    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.354920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.354904    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.368519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.367572    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd44c\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.391880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.391804    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.393079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.393048    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.404953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.396949    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.404953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.397003    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5s6\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.404953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.402860    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token\") pod \"e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") " pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.415972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.415939    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"]
Nov 02 18:21:31.417890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.417859    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-ltw25"]
Nov 02 18:21:31.491925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.491180    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:31.504972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.504942    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:31.548932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.548900    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d\" (UniqueName: \"kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d\") pod \"1e49cc75-fea8-4d97-b290-7922411562b3\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") "
Nov 02 18:21:31.549367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.549202    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca\") pod \"1e49cc75-fea8-4d97-b290-7922411562b3\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") "
Nov 02 18:21:31.552872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.552819    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca\") pod \"1e49cc75-fea8-4d97-b290-7922411562b3\" (UID: \"1e49cc75-fea8-4d97-b290-7922411562b3\") "
Nov 02 18:21:31.557014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.556985    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:31.558225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.558058    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca" pod "1e49cc75-fea8-4d97-b290-7922411562b3" (UID: "1e49cc75-fea8-4d97-b290-7922411562b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:31.578078 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.577793    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" path="/var/lib/kubelet/pods/db2d4865-b81a-4d68-9dde-1bc29f58bde7/volumes"
Nov 02 18:21:31.579044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.579020    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff95a5d7-5e13-4987-9a9f-6c10ced46438" path="/var/lib/kubelet/pods/ff95a5d7-5e13-4987-9a9f-6c10ced46438/volumes"
Nov 02 18:21:31.581550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.581524    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca" pod "1e49cc75-fea8-4d97-b290-7922411562b3" (UID: "1e49cc75-fea8-4d97-b290-7922411562b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:31.584900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.584863    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d" pod "1e49cc75-fea8-4d97-b290-7922411562b3" (UID: "1e49cc75-fea8-4d97-b290-7922411562b3"). InnerVolumeSpecName "kube-api-access-6sg7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:31.658833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.658721    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.658833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.658782    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sg7d\" (UniqueName: \"kubernetes.io/projected/1e49cc75-fea8-4d97-b290-7922411562b3-kube-api-access-6sg7d\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.658833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.658799    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e49cc75-fea8-4d97-b290-7922411562b3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:31.918649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.918599    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"]
Nov 02 18:21:31.926972 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:21:31.926925    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07d1a73_10bb_4dfe_9a07_b5e81ccc6d83.slice/crio-b4c600adcac71572cfb4602adfd2898b44feb2a1447027f48e33002cb8527028 WatchSource:0}: Error finding container b4c600adcac71572cfb4602adfd2898b44feb2a1447027f48e33002cb8527028: Status 404 returned error can't find the container with id b4c600adcac71572cfb4602adfd2898b44feb2a1447027f48e33002cb8527028
Nov 02 18:21:31.928768 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.928506    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:31.928768 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.928601    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:31.928768 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.928643    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:31.933398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.931335    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"]
Nov 02 18:21:31.941226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.937488    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:31.941226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.937590    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:31.941226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:31.937627    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:32.084969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.084200    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" event={"ID":"ad488816-a94d-4505-b09b-77abcee964e0","Type":"ContainerStarted","Data":"07deff1039bf6207e6a7bff8b741cd460a4c32d0142ba82a075a23475524b4c0"}
Nov 02 18:21:32.088681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.088451    5042 generic.go:356] "Generic (PLEG): container finished" podID="8b63ef55-620f-40bc-acef-2939dbad9279" containerID="a9d28fa02ee73fbae4993a6b58af4f3851ad467579cc8f0d1dbe422c4eb987d3" exitCode=0
Nov 02 18:21:32.088681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.088551    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" event={"ID":"8b63ef55-620f-40bc-acef-2939dbad9279","Type":"ContainerDied","Data":"a9d28fa02ee73fbae4993a6b58af4f3851ad467579cc8f0d1dbe422c4eb987d3"}
Nov 02 18:21:32.088681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.088607    5042 scope.go:122] "RemoveContainer" containerID="b21ac7e159f5a5e29d20dbf3cf7e6e30d5dab7fe31700009c79885b6d044984a"
Nov 02 18:21:32.092077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.092012    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" event={"ID":"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83","Type":"ContainerStarted","Data":"b4c600adcac71572cfb4602adfd2898b44feb2a1447027f48e33002cb8527028"}
Nov 02 18:21:32.095260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.095237    5042 generic.go:356] "Generic (PLEG): container finished" podID="aa199220-32ef-4eea-a688-05f166f14d39" containerID="6873572cd2f7d3b465cc055bc615a163e703bad331990bfd253a0cf43e5a0f9b" exitCode=0
Nov 02 18:21:32.095364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.095338    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" event={"ID":"aa199220-32ef-4eea-a688-05f166f14d39","Type":"ContainerDied","Data":"6873572cd2f7d3b465cc055bc615a163e703bad331990bfd253a0cf43e5a0f9b"}
Nov 02 18:21:32.098916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.098703    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm" event={"ID":"1e49cc75-fea8-4d97-b290-7922411562b3","Type":"ContainerDied","Data":"4249203ca775de728a82acd8221d26cab3af94eb07306be03c57609a1f467b6f"}
Nov 02 18:21:32.098916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.098741    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4249203ca775de728a82acd8221d26cab3af94eb07306be03c57609a1f467b6f"
Nov 02 18:21:32.098916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.098812    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"
Nov 02 18:21:32.270576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.270544    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.401460    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.401578    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.401642    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.401681    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.401729    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.401961    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.402085    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.402155    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.404921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.402224    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554\") pod \"8b63ef55-620f-40bc-acef-2939dbad9279\" (UID: \"8b63ef55-620f-40bc-acef-2939dbad9279\") "
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.457079    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.478001    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.478484    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.503062    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.503169    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.508647    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.508682    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.508696    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b63ef55-620f-40bc-acef-2939dbad9279-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.508709    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.556730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.508722    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.664774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.664054    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:32.679883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.674995    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:32.746640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.738222    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:32.746640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.741783    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554" pod "8b63ef55-620f-40bc-acef-2939dbad9279" (UID: "8b63ef55-620f-40bc-acef-2939dbad9279"). InnerVolumeSpecName "kube-api-access-vs554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:32.746640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.745967    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs554\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-kube-api-access-vs554\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.746640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.746001    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b63ef55-620f-40bc-acef-2939dbad9279-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.746640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.746015    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b63ef55-620f-40bc-acef-2939dbad9279-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.746640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.746031    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b63ef55-620f-40bc-acef-2939dbad9279-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.842980    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"]
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.845876    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.845903    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.845932    5042 cpu_manager.go:414] "containerMap: removing container" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.845940    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2d4865-b81a-4d68-9dde-1bc29f58bde7" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.845974    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1e49cc75-fea8-4d97-b290-7922411562b3" containerName="revoke-keys"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.845982    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e49cc75-fea8-4d97-b290-7922411562b3" containerName="revoke-keys"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.846030    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.846039    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.846259    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e49cc75-fea8-4d97-b290-7922411562b3" containerName="revoke-keys"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.846279    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.846298    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" containerName="istio-proxy"
Nov 02 18:21:32.868648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.861652    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:32.881450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.869627    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:21:32.892091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.891954    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"]
Nov 02 18:21:32.939918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.939697    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"]
Nov 02 18:21:32.945043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.941344    5042 cpu_manager.go:414] "containerMap: removing container" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy"
Nov 02 18:21:32.945043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.941369    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy"
Nov 02 18:21:32.945043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.941681    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa199220-32ef-4eea-a688-05f166f14d39" containerName="istio-proxy"
Nov 02 18:21:32.950181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.949211    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:32.950181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:32.949631    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"]
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005067    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005165    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005218    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005279    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005315    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005459    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005578    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005624    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.005783    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token\") pod \"aa199220-32ef-4eea-a688-05f166f14d39\" (UID: \"aa199220-32ef-4eea-a688-05f166f14d39\") "
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.007790    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.007982    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008061    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008128    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008154    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008329    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008376    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008499    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvpjw\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.008561    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.009221    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.009502    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.009693    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.009916    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.010537    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.010821    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:33.010932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.010950    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.097466    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "kube-api-access-lk2fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117081    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117160    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117204    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117250    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117303    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117391    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117431    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvh6t\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117477    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvpjw\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117549    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.117573    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118231    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118319    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118463    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118537    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118577    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118655    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118737    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118765    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118963    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2fc\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-kube-api-access-lk2fc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118980    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.118995    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.119009    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa199220-32ef-4eea-a688-05f166f14d39-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.119026    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.119040    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.119057    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa199220-32ef-4eea-a688-05f166f14d39-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.119071    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa199220-32ef-4eea-a688-05f166f14d39-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.128586    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo" pod "aa199220-32ef-4eea-a688-05f166f14d39" (UID: "aa199220-32ef-4eea-a688-05f166f14d39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.158584    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.158941    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.162264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.159980    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.181303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.164206    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.181303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.164571    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.181303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.165238    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.181303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.171737    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.181303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.174892    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvpjw\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.181303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.178425    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" event={"ID":"ad488816-a94d-4505-b09b-77abcee964e0","Type":"ContainerStarted","Data":"22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a"}
Nov 02 18:21:33.182214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.181800    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm" event={"ID":"8b63ef55-620f-40bc-acef-2939dbad9279","Type":"ContainerDied","Data":"af9119d57f519fa8c0ed2bf2da1e5e1d69c916ac70b673c7169598d730e0325f"}
Nov 02 18:21:33.182214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.181955    5042 scope.go:122] "RemoveContainer" containerID="a9d28fa02ee73fbae4993a6b58af4f3851ad467579cc8f0d1dbe422c4eb987d3"
Nov 02 18:21:33.182214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.182084    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"
Nov 02 18:21:33.198564 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.197391    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" event={"ID":"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83","Type":"ContainerStarted","Data":"e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"}
Nov 02 18:21:33.201178 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.201141    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.205140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.205013    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv" event={"ID":"aa199220-32ef-4eea-a688-05f166f14d39","Type":"ContainerDied","Data":"182d1b29a1cc9e33be29cb9354099b8f4d5c945cdc53107c2e361a78a5ec38da"}
Nov 02 18:21:33.208867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.205373    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"
Nov 02 18:21:33.214251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.214198    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:33.221394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.218189    5042 scope.go:122] "RemoveContainer" containerID="6873572cd2f7d3b465cc055bc615a163e703bad331990bfd253a0cf43e5a0f9b"
Nov 02 18:21:33.221394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.220034    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podStartSLOduration=170.220015271 podStartE2EDuration="2m50.220015271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:33.216534601 +0000 UTC m=+2905.795943987" watchObservedRunningTime="2026-11-02 18:21:33.220015271 +0000 UTC m=+2905.799424660"
Nov 02 18:21:33.221394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221071    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221304    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221365    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221454    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221540    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221569    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvh6t\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221644    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.221816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221682    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.221828    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.222327    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa199220-32ef-4eea-a688-05f166f14d39-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.223052    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.223220    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.223375    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.223754    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.228019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.225062    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.231497 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.230610    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.231497 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.231198    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.243306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.243211    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.249655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.249622    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvh6t\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t\") pod \"e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") " pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.252057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.252006    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"]
Nov 02 18:21:33.261241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.261213    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-4gtzm"]
Nov 02 18:21:33.282280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.282209    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podStartSLOduration=123.282186876 podStartE2EDuration="2m3.282186876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:33.269930117 +0000 UTC m=+2905.849339502" watchObservedRunningTime="2026-11-02 18:21:33.282186876 +0000 UTC m=+2905.861596263"
Nov 02 18:21:33.297398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.297331    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"]
Nov 02 18:21:33.303657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.303617    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:33.306053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.305902    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-bx7wv"]
Nov 02 18:21:33.508991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.508781    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:33.509316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.509289    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:33.509400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.509347    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:33.565978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.565571    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b63ef55-620f-40bc-acef-2939dbad9279" path="/var/lib/kubelet/pods/8b63ef55-620f-40bc-acef-2939dbad9279/volumes"
Nov 02 18:21:33.567172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.567095    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa199220-32ef-4eea-a688-05f166f14d39" path="/var/lib/kubelet/pods/aa199220-32ef-4eea-a688-05f166f14d39/volumes"
Nov 02 18:21:33.568074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.568054    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:33.568207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.568196    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:33.599668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.599579    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"]
Nov 02 18:21:33.599965 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:21:33.599935    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f2101f_bb0b_4e97_aee9_263faee9ad91.slice/crio-2d8bfb24b78a6bcfababae5e0f6a84bc335e70bb4d08dfc3c50f662ac7af97de WatchSource:0}: Error finding container 2d8bfb24b78a6bcfababae5e0f6a84bc335e70bb4d08dfc3c50f662ac7af97de: Status 404 returned error can't find the container with id 2d8bfb24b78a6bcfababae5e0f6a84bc335e70bb4d08dfc3c50f662ac7af97de
Nov 02 18:21:33.606221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.606178    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:33.606452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.606430    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:33.606586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.606570    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:33.801470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.801420    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:33.801632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.801579    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:33.801703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.801663    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:21:33.822537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:33.812355    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"]
Nov 02 18:21:34.233305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.233261    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" event={"ID":"b5f2101f-bb0b-4e97-aee9-263faee9ad91","Type":"ContainerStarted","Data":"6d0261cbdf93548b7254502b814755e9e0dfa2104e9066291c1c4e114fdf5364"}
Nov 02 18:21:34.233976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.233944    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" event={"ID":"b5f2101f-bb0b-4e97-aee9-263faee9ad91","Type":"ContainerStarted","Data":"2d8bfb24b78a6bcfababae5e0f6a84bc335e70bb4d08dfc3c50f662ac7af97de"}
Nov 02 18:21:34.237660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.237634    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" event={"ID":"b42a4cf5-0cc5-4491-9b14-e1b02cd96498","Type":"ContainerStarted","Data":"9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b"}
Nov 02 18:21:34.237806 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.237681    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" event={"ID":"b42a4cf5-0cc5-4491-9b14-e1b02cd96498","Type":"ContainerStarted","Data":"2d2bb355bd312ea318709296cc82cb201f9d43cc3cb0fa36325af8e4d5fda3d6"}
Nov 02 18:21:34.239729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.239707    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:34.240823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.240802    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:21:34.257358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.257295    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podStartSLOduration=111.257275523 podStartE2EDuration="1m51.257275523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:34.255177651 +0000 UTC m=+2906.834587073" watchObservedRunningTime="2026-11-02 18:21:34.257275523 +0000 UTC m=+2906.836684909"
Nov 02 18:21:34.306467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.306378    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:34.366374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.365780    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podStartSLOduration=111.36575856 podStartE2EDuration="1m51.36575856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:34.331511446 +0000 UTC m=+2906.910920833" watchObservedRunningTime="2026-11-02 18:21:34.36575856 +0000 UTC m=+2906.945167947"
Nov 02 18:21:34.545450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.545393    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:34.545786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.545737    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:34.549233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.549198    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"]
Nov 02 18:21:34.550829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:34.549377    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" containerID="cri-o://163a1e0a44f44cdf8289769a70ccbac735501053f844f786372be9b61a659a12" gracePeriod=30
Nov 02 18:21:35.232036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.231941    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:35.234066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.233690    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:35.234066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.233749    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:35.326332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.325966    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:35.326332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.326037    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:35.507476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.507382    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:35.507766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:35.507734    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:36.216313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.216275    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:36.219056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.216540    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:36.400012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.367662    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:36.400012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.367729    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:36.504050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.503944    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 18:21:36.506447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.506416    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:36.506696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:36.506668    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:37.277867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:37.243695    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:37.277867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:37.243767    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:37.353401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:37.353363    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:37.382151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:37.356936    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:37.527940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:37.523961    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:37.527940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:37.524029    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:38.037881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.034819    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Nov 02 18:21:38.167548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.166050    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:38.223865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.222988    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:38.223865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.223073    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:38.313919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.313702    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:38.313919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.313779    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:38.365995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.361078    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:38.472866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.471821    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:38.555877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.540355    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:38.598919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.596107    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:38.598919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.596204    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:38.875468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:38.875333    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:39.213009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.212963    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:39.213536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.213508    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:21:39.215379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.215353    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:39.229297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.229236    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:39.343439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.319979    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:39.343439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.320055    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:39.409825 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.409765    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:39.438869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.410602    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 18:21:39.684939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.486472    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:39.684939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.487175    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:39.684939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.492007    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:21:39.684939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.670034    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:39.684939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:39.670108    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:40.114867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.110012    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:40.143892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.136261    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"]
Nov 02 18:21:40.143892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.136561    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" containerID="cri-o://110fe61ad703b6513d309023bc8955ced31d09ee27bc2408e6bc438e5106986f" gracePeriod=30
Nov 02 18:21:40.155178 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.151862    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:21:40.170047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.169980    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:40.240153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.237151    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:40.240153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.237223    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:40.262423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.262288    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:40.421911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.419352    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:21:40.437519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.437485    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:40.439002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.438969    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:40.484960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.456214    5042 generic.go:356] "Generic (PLEG): container finished" podID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerID="163a1e0a44f44cdf8289769a70ccbac735501053f844f786372be9b61a659a12" exitCode=0
Nov 02 18:21:40.485400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.456243    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" event={"ID":"1146d135-49fd-4c59-bf15-ea822c104e1a","Type":"ContainerDied","Data":"163a1e0a44f44cdf8289769a70ccbac735501053f844f786372be9b61a659a12"}
Nov 02 18:21:40.500033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.499980    5042 scope.go:122] "RemoveContainer" containerID="05cadd8ec1d38d2b6b1f5335d0474fa22a4516f0d06510ef7841cb73393706c3"
Nov 02 18:21:40.638047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.638007    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:40.677986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.638331    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:40.760879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:40.680935    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 02 18:21:41.302005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.301955    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:41.366749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.366692    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:41.367112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.367057    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:41.434180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.434106    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:41.611783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.573188    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:41.611783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.596062    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:41.611783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.596122    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:41.611783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.608330    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:41.611783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:41.608385    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:42.138464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.138421    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 18:21:42.211875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.203201    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:42.223380 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.218475    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:42.310867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.307671    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:42.470895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.346971    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:42.470895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.469523    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:42.470895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.396082    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:42.470895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.469579    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:42.560862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.523221    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:42.661018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.623952    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 18:21:42.818956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.816917    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body=
Nov 02 18:21:42.818956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.816993    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused"
Nov 02 18:21:42.908239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.908177    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:42.910347 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:42.910305    5042 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.366s"
Nov 02 18:21:42.910719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:42.910702    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:21:43.450151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.450118    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.553073    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.260690    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.187853    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.265292    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.556298    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.454805    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused" start-of-body=
Nov 02 18:21:43.562867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.556359    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.68:15021/healthz/ready\": dial tcp 10.132.1.68:15021: connect: connection refused"
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.636927    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.636978    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637007    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637098    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637173    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637225    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637343    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637382    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.650865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.637450    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy\") pod \"1146d135-49fd-4c59-bf15-ea822c104e1a\" (UID: \"1146d135-49fd-4c59-bf15-ea822c104e1a\") "
Nov 02 18:21:43.669203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.669156    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:21:43.684321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.684278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:43.685513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.685417    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:43.762226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.762140    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:43.855222 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.855184    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:43.856068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.856042    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:43.856268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.856249    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:43.862506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.862468    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:43.879613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.879557    5042 patch_prober.go:29] interesting pod/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:21:43.879748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.879651    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:43.885373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.885328    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "kube-api-access-mx8nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890680    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890728    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890745    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890761    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890776    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx8nr\" (UniqueName: \"kubernetes.io/projected/1146d135-49fd-4c59-bf15-ea822c104e1a-kube-api-access-mx8nr\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890797    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890812    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.895136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.890875    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1146d135-49fd-4c59-bf15-ea822c104e1a-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:43.914691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:43.914648    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert" pod "1146d135-49fd-4c59-bf15-ea822c104e1a" (UID: "1146d135-49fd-4c59-bf15-ea822c104e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:44.049629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.047981    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1146d135-49fd-4c59-bf15-ea822c104e1a-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:44.062943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.056966    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:21:44.062943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.057071    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 18:21:44.062943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.057098    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76" event={"ID":"1146d135-49fd-4c59-bf15-ea822c104e1a","Type":"ContainerDied","Data":"60afe0e729819f3824d09c20e942e42fcd7e1f8e92a0c9677034f642086ec718"}
Nov 02 18:21:44.062943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.057242    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:44.062943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.057308    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:21:44.070990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.068689    5042 patch_prober.go:29] interesting pod/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:21:44.070990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.068796    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:21:44.071462 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.071438    5042 scope.go:122] "RemoveContainer" containerID="163a1e0a44f44cdf8289769a70ccbac735501053f844f786372be9b61a659a12"
Nov 02 18:21:44.072669 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.072204    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:44.075300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.074505    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:21:44.221868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.220886    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body=
Nov 02 18:21:44.221868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.220951    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused"
Nov 02 18:21:44.232906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.231885    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"]
Nov 02 18:21:44.232906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.232203    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy" containerID="cri-o://a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4" gracePeriod=30
Nov 02 18:21:44.244531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.244494    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"]
Nov 02 18:21:44.245083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.245010    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy" containerID="cri-o://54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94" gracePeriod=30
Nov 02 18:21:44.581318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.581273    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"
Nov 02 18:21:44.611409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.603553    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:44.611409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.604550    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:44.611409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.607572    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:21:44.619086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.619053    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"]
Nov 02 18:21:44.627031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.626981    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-a37cf0-openshift-default-d9d86696c-w2m76"]
Nov 02 18:21:44.768411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:44.768369    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"]
Nov 02 18:21:45.695863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:45.690444    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" path="/var/lib/kubelet/pods/1146d135-49fd-4c59-bf15-ea822c104e1a/volumes"
Nov 02 18:21:45.768864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:45.765459    5042 generic.go:356] "Generic (PLEG): container finished" podID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerID="110fe61ad703b6513d309023bc8955ced31d09ee27bc2408e6bc438e5106986f" exitCode=0
Nov 02 18:21:45.768864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:45.766761    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" event={"ID":"b9d03ddb-5a13-4516-bdbf-8229f0caec89","Type":"ContainerDied","Data":"110fe61ad703b6513d309023bc8955ced31d09ee27bc2408e6bc438e5106986f"}
Nov 02 18:21:46.727922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:46.719529    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"]
Nov 02 18:21:46.817867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:46.790220    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf" podUID="033e412d-b49f-4076-adec-af98cec94f3f" containerName="maas-api" containerID="cri-o://bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86" gracePeriod=30
Nov 02 18:21:46.875866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:46.869038    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:21:46.875866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:46.869103    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:21:46.920919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:46.879746    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:21:46.998820 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:46.998053    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d03ddb_5a13_4516_bdbf_8229f0caec89.slice/crio-conmon-110fe61ad703b6513d309023bc8955ced31d09ee27bc2408e6bc438e5106986f.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:21:47.049211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.045978    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 18:21:47.256030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.255907    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:47.256612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.256584    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:47.258191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.258140    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:21:47.343908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.342078    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 18:21:47.558253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558202    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558330    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558391    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558429    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558520    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558624    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558685    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558816    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.558921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.558863    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket\") pod \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\" (UID: \"b9d03ddb-5a13-4516-bdbf-8229f0caec89\") "
Nov 02 18:21:47.592880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.586401    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:47.615970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.615873    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "kube-api-access-6qrcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:47.617980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.617936    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:47.618992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.618689    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:47.618992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.618892    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:47.621865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.621571    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:47.623074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.623048    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:47.628032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.627801    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:47.628213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.628186    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket" pod "b9d03ddb-5a13-4516-bdbf-8229f0caec89" (UID: "b9d03ddb-5a13-4516-bdbf-8229f0caec89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:47.639870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.638908    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.661891    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.661926    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.661942    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.661958    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.661972    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.661986    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.662000    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qrcm\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-kube-api-access-6qrcm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.662013    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9d03ddb-5a13-4516-bdbf-8229f0caec89-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.663889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.662026    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9d03ddb-5a13-4516-bdbf-8229f0caec89-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.763410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.763369    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs\") pod \"033e412d-b49f-4076-adec-af98cec94f3f\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") "
Nov 02 18:21:47.763925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.763493    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt\" (UniqueName: \"kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt\") pod \"033e412d-b49f-4076-adec-af98cec94f3f\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") "
Nov 02 18:21:47.763925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.763763    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca\") pod \"033e412d-b49f-4076-adec-af98cec94f3f\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") "
Nov 02 18:21:47.764058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.763942    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls\") pod \"033e412d-b49f-4076-adec-af98cec94f3f\" (UID: \"033e412d-b49f-4076-adec-af98cec94f3f\") "
Nov 02 18:21:47.764206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.764176    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca" pod "033e412d-b49f-4076-adec-af98cec94f3f" (UID: "033e412d-b49f-4076-adec-af98cec94f3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:47.764690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.764669    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033e412d-b49f-4076-adec-af98cec94f3f-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.765933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.765906    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs" pod "033e412d-b49f-4076-adec-af98cec94f3f" (UID: "033e412d-b49f-4076-adec-af98cec94f3f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:47.766052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.765935    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt" pod "033e412d-b49f-4076-adec-af98cec94f3f" (UID: "033e412d-b49f-4076-adec-af98cec94f3f"). InnerVolumeSpecName "kube-api-access-gmgzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:47.766105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.766072    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls" pod "033e412d-b49f-4076-adec-af98cec94f3f" (UID: "033e412d-b49f-4076-adec-af98cec94f3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:47.866127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.866037    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmgzt\" (UniqueName: \"kubernetes.io/projected/033e412d-b49f-4076-adec-af98cec94f3f-kube-api-access-gmgzt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.866127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.866069    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.866127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.866081    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033e412d-b49f-4076-adec-af98cec94f3f-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:47.978072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.978033    5042 generic.go:356] "Generic (PLEG): container finished" podID="033e412d-b49f-4076-adec-af98cec94f3f" containerID="bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86" exitCode=0
Nov 02 18:21:47.978292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.978085    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"
Nov 02 18:21:47.978292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.978126    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf" event={"ID":"033e412d-b49f-4076-adec-af98cec94f3f","Type":"ContainerDied","Data":"bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86"}
Nov 02 18:21:47.978292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.978195    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf" event={"ID":"033e412d-b49f-4076-adec-af98cec94f3f","Type":"ContainerDied","Data":"185f6e02c254df6ccbcf310d7c806b97d933c374f57337ba656de681a547ea0f"}
Nov 02 18:21:47.978292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.978214    5042 scope.go:122] "RemoveContainer" containerID="bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86"
Nov 02 18:21:47.980474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.980452    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"
Nov 02 18:21:47.980610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.980499    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x" event={"ID":"b9d03ddb-5a13-4516-bdbf-8229f0caec89","Type":"ContainerDied","Data":"3064864cc7b6d5553a80f0e0988393c25af84ea6433d6f9eadcafbafa4c09321"}
Nov 02 18:21:47.994439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.993960    5042 scope.go:122] "RemoveContainer" containerID="bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86"
Nov 02 18:21:47.995234 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:47.995192    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86\": container with ID starting with bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86 not found: ID does not exist" containerID="bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86"
Nov 02 18:21:47.995337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.995235    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86"} err="failed to get container status \"bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86\": rpc error: code = NotFound desc = could not find container \"bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86\": container with ID starting with bcc3307a033c3349f5117d8065b17c8b5fd584944f1accada2ba61d3c2cd8a86 not found: ID does not exist"
Nov 02 18:21:47.995337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:47.995259    5042 scope.go:122] "RemoveContainer" containerID="110fe61ad703b6513d309023bc8955ced31d09ee27bc2408e6bc438e5106986f"
Nov 02 18:21:48.008681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:48.008652    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"]
Nov 02 18:21:48.015924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:48.015434    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b97c6-openshift-default-bc5d59c59-57d2x"]
Nov 02 18:21:48.021355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:48.021327    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"]
Nov 02 18:21:48.026494 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:48.026467    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-312f21-6f974cfb7-xlwhf"]
Nov 02 18:21:48.542481 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:48.542444    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:21:49.554774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.553799    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033e412d-b49f-4076-adec-af98cec94f3f" path="/var/lib/kubelet/pods/033e412d-b49f-4076-adec-af98cec94f3f/volumes"
Nov 02 18:21:49.554774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.554676    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" path="/var/lib/kubelet/pods/b9d03ddb-5a13-4516-bdbf-8229f0caec89/volumes"
Nov 02 18:21:49.623584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.623554    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.672967    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673059    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673171    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673308    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673379    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673430    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673462    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673572    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.685867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.673642    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo\") pod \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\" (UID: \"dd705464-df0f-4d2f-bb88-3f9ef9057ad6\") "
Nov 02 18:21:49.736231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.736187    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.737530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.737485    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.739059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.739024    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.741788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.741721    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.743693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.743646    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:49.753940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.753754    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.753940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.753908    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "kube-api-access-xh7bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:49.762207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.761988    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:49.762207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.762171    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token" pod "dd705464-df0f-4d2f-bb88-3f9ef9057ad6" (UID: "dd705464-df0f-4d2f-bb88-3f9ef9057ad6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:49.776931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.772981    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" containerName="payload-processing" containerID="cri-o://6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd" gracePeriod=25
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792226    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792264    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792279    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7bj\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-kube-api-access-xh7bj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792295    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792309    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792322    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792336    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792350    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.792918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.792363    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd705464-df0f-4d2f-bb88-3f9ef9057ad6-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.801252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.801178    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:21:49.893624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.893588    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.893985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.893919    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.894129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.894010    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.894129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.894064    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.894252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.894175    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.894364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.894348    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.894910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.894765    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.895175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.895145    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.895325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.895301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.895430 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.895407    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket\") pod \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\" (UID: \"a8b919ab-89b4-492b-a4fa-532cfb80bc48\") "
Nov 02 18:21:49.895932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.895701    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:49.895932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.895798    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.895932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.895896    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.896190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.896102    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.896926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.896902    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.897016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.896930    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.897016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.896944    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.897016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.896961    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.897016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.896975    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.901440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.901411    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "kube-api-access-7v955". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:49.902752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.902731    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:49.903530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.903503    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:49.905987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.905959    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token" pod "a8b919ab-89b4-492b-a4fa-532cfb80bc48" (UID: "a8b919ab-89b4-492b-a4fa-532cfb80bc48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:49.998732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.998685    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.998732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.998728    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.999085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.998745    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b919ab-89b4-492b-a4fa-532cfb80bc48-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:49.999085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:49.998761    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v955\" (UniqueName: \"kubernetes.io/projected/a8b919ab-89b4-492b-a4fa-532cfb80bc48-kube-api-access-7v955\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:50.011402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.011363    5042 generic.go:356] "Generic (PLEG): container finished" podID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerID="54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94" exitCode=0
Nov 02 18:21:50.011611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.011409    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"
Nov 02 18:21:50.011611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.011442    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" event={"ID":"a8b919ab-89b4-492b-a4fa-532cfb80bc48","Type":"ContainerDied","Data":"54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94"}
Nov 02 18:21:50.011611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.011501    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24" event={"ID":"a8b919ab-89b4-492b-a4fa-532cfb80bc48","Type":"ContainerDied","Data":"ad7c0f40408d2e9e0c0314576312efb9d2c926d169d823d331cb2e3101d42dfd"}
Nov 02 18:21:50.011611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.011522    5042 scope.go:122] "RemoveContainer" containerID="54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94"
Nov 02 18:21:50.014023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.013996    5042 generic.go:356] "Generic (PLEG): container finished" podID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerID="a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4" exitCode=0
Nov 02 18:21:50.014172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.014066    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"
Nov 02 18:21:50.014172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.014110    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" event={"ID":"dd705464-df0f-4d2f-bb88-3f9ef9057ad6","Type":"ContainerDied","Data":"a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4"}
Nov 02 18:21:50.014311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.014186    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8" event={"ID":"dd705464-df0f-4d2f-bb88-3f9ef9057ad6","Type":"ContainerDied","Data":"d8c4337c2eff5282c51d2f50d28754b54ddc03476ace1a61c15fcffa82ff224c"}
Nov 02 18:21:50.026378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.026351    5042 scope.go:122] "RemoveContainer" containerID="54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94"
Nov 02 18:21:50.026733 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:50.026711    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94\": container with ID starting with 54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94 not found: ID does not exist" containerID="54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94"
Nov 02 18:21:50.026883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.026741    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94"} err="failed to get container status \"54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94\": rpc error: code = NotFound desc = could not find container \"54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94\": container with ID starting with 54e32387643e6246723b70225a3fbe69ddfc98f3bb33ff03bce42b4028e7fe94 not found: ID does not exist"
Nov 02 18:21:50.026883 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.026757    5042 scope.go:122] "RemoveContainer" containerID="a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4"
Nov 02 18:21:50.044612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.044587    5042 scope.go:122] "RemoveContainer" containerID="c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"
Nov 02 18:21:50.047870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.047814    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"]
Nov 02 18:21:50.053420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.053341    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-97z24"]
Nov 02 18:21:50.069741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.069601    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"]
Nov 02 18:21:50.075238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.075208    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-42d786-openshift-default-776544c777-x7ff8"]
Nov 02 18:21:50.077623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.077264    5042 scope.go:122] "RemoveContainer" containerID="a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4"
Nov 02 18:21:50.085858 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:50.077738    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4\": container with ID starting with a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4 not found: ID does not exist" containerID="a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4"
Nov 02 18:21:50.085858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.077790    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4"} err="failed to get container status \"a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4\": rpc error: code = NotFound desc = could not find container \"a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4\": container with ID starting with a8d4892adaf0b7b1f00a97541c593189cca9afa5c5eb8fe7339a0d68e4956bc4 not found: ID does not exist"
Nov 02 18:21:50.085858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.077821    5042 scope.go:122] "RemoveContainer" containerID="c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"
Nov 02 18:21:50.085858 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:50.082801    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312\": container with ID starting with c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312 not found: ID does not exist" containerID="c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"
Nov 02 18:21:50.085858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:50.082832    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312"} err="failed to get container status \"c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312\": rpc error: code = NotFound desc = could not find container \"c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312\": container with ID starting with c99db867758b7364bf756eb3fba4e2896471a1889c2a47c460d840bd75623312 not found: ID does not exist"
Nov 02 18:21:51.575914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:51.575878    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" path="/var/lib/kubelet/pods/a8b919ab-89b4-492b-a4fa-532cfb80bc48/volumes"
Nov 02 18:21:51.576668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:51.576645    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" path="/var/lib/kubelet/pods/dd705464-df0f-4d2f-bb88-3f9ef9057ad6/volumes"
Nov 02 18:21:52.223387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:52.223343    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"]
Nov 02 18:21:52.223666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:52.223637    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy" containerID="cri-o://1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220" gracePeriod=30
Nov 02 18:21:57.569486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.569452    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:21:57.739448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739345    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.739448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739397    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.739741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739460    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.739741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739517    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.739741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739546    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.739741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739597    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.739741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739692    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.740011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.739777    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.740062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.740011    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy\") pod \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\" (UID: \"f1d3f0cc-9bff-48aa-a233-c572e9097b13\") "
Nov 02 18:21:57.740693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.740664    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:57.741001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.740982    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.741678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.741228    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:57.741678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.741393    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:57.741678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.741441    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:57.741678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.741448    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:57.743312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.743286    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:21:57.743476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.743459    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:57.743578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.743559    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "kube-api-access-slkbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:57.743693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.743672    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo" pod "f1d3f0cc-9bff-48aa-a233-c572e9097b13" (UID: "f1d3f0cc-9bff-48aa-a233-c572e9097b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:21:57.855931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.845256    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.855931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.845297    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slkbm\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-kube-api-access-slkbm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.855931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.845313    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.856441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.856330    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.856441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.856375    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.856441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.856393    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.856441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.856406    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:57.856441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:57.856422    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1d3f0cc-9bff-48aa-a233-c572e9097b13-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:21:58.213065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.213030    5042 generic.go:356] "Generic (PLEG): container finished" podID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerID="1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220" exitCode=0
Nov 02 18:21:58.213265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.213089    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"
Nov 02 18:21:58.213265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.213091    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" event={"ID":"f1d3f0cc-9bff-48aa-a233-c572e9097b13","Type":"ContainerDied","Data":"1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220"}
Nov 02 18:21:58.213265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.213230    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr" event={"ID":"f1d3f0cc-9bff-48aa-a233-c572e9097b13","Type":"ContainerDied","Data":"6e3ebb7b63f720187e60cf5e6a0862073063647b0beee996ebc1c89cbc7c3557"}
Nov 02 18:21:58.213265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.213261    5042 scope.go:122] "RemoveContainer" containerID="1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220"
Nov 02 18:21:58.228911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.228891    5042 scope.go:122] "RemoveContainer" containerID="1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220"
Nov 02 18:21:58.229271 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:21:58.229250    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220\": container with ID starting with 1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220 not found: ID does not exist" containerID="1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220"
Nov 02 18:21:58.229338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.229284    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220"} err="failed to get container status \"1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220\": rpc error: code = NotFound desc = could not find container \"1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220\": container with ID starting with 1c4ed28b4ab818e0b0c91eab468c592b79393f2ae3120a179948d4086f13a220 not found: ID does not exist"
Nov 02 18:21:58.246200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.246169    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"]
Nov 02 18:21:58.267782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.267741    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-312f21-openshift-default-69765ddd69-8cvmr"]
Nov 02 18:21:58.551738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.551640    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"]
Nov 02 18:21:58.778966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:58.778927    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:21:59.547468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:21:59.547435    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" path="/var/lib/kubelet/pods/f1d3f0cc-9bff-48aa-a233-c572e9097b13/volumes"
Nov 02 18:22:03.557601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:03.557546    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerName="payload-processing" containerID="cri-o://a94f48dd752e7481b78be34df1fe6772e02e86d4494d2412e85400e7753b4aa5" gracePeriod=25
Nov 02 18:22:08.777854 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:08.777811    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:22:10.198426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.198384    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:22:10.240616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.240581    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:22:10.336611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.336570    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz\" (UniqueName: \"kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz\") pod \"dd745b23-0598-4e19-9537-3ed5c422891e\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") "
Nov 02 18:22:10.336938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.336915    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume\") pod \"dd745b23-0598-4e19-9537-3ed5c422891e\" (UID: \"dd745b23-0598-4e19-9537-3ed5c422891e\") "
Nov 02 18:22:10.337523 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.337488    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume" pod "dd745b23-0598-4e19-9537-3ed5c422891e" (UID: "dd745b23-0598-4e19-9537-3ed5c422891e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:22:10.337687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.337670    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd745b23-0598-4e19-9537-3ed5c422891e-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:10.339716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.339688    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz" pod "dd745b23-0598-4e19-9537-3ed5c422891e" (UID: "dd745b23-0598-4e19-9537-3ed5c422891e"). InnerVolumeSpecName "kube-api-access-8nkzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:22:10.374936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.374897    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"]
Nov 02 18:22:10.375727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375710    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy"
Nov 02 18:22:10.375727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375727    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375741    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375747    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375758    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375764    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375773    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375778    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375791    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375795    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375804    5042 cpu_manager.go:414] "containerMap: removing container" podUID="033e412d-b49f-4076-adec-af98cec94f3f" containerName="maas-api"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375808    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033e412d-b49f-4076-adec-af98cec94f3f" containerName="maas-api"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375827    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" containerName="payload-processing"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375832    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" containerName="payload-processing"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375866    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375874    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375892    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy"
Nov 02 18:22:10.375886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.375898    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376027    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376034    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1146d135-49fd-4c59-bf15-ea822c104e1a" containerName="istio-proxy"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376043    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d03ddb-5a13-4516-bdbf-8229f0caec89" containerName="istio-proxy"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376050    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b919ab-89b4-492b-a4fa-532cfb80bc48" containerName="istio-proxy"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376057    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376062    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="033e412d-b49f-4076-adec-af98cec94f3f" containerName="maas-api"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376069    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" containerName="payload-processing"
Nov 02 18:22:10.377353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.376076    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d3f0cc-9bff-48aa-a233-c572e9097b13" containerName="istio-proxy"
Nov 02 18:22:10.380093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.380072    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.387073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.387044    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:22:10.387982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.387955    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:22:10.391777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.391749    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"]
Nov 02 18:22:10.439683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.439317    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nkzz\" (UniqueName: \"kubernetes.io/projected/dd745b23-0598-4e19-9537-3ed5c422891e-kube-api-access-8nkzz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:10.540607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.540570    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4hj\" (UniqueName: \"kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.540789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.540618    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.540789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.540732    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.585600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.585566    5042 generic.go:356] "Generic (PLEG): container finished" podID="dd745b23-0598-4e19-9537-3ed5c422891e" containerID="6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd" exitCode=0
Nov 02 18:22:10.585809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.585626    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"
Nov 02 18:22:10.585809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.585644    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" event={"ID":"dd745b23-0598-4e19-9537-3ed5c422891e","Type":"ContainerDied","Data":"6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd"}
Nov 02 18:22:10.585809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.585716    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn" event={"ID":"dd745b23-0598-4e19-9537-3ed5c422891e","Type":"ContainerDied","Data":"95534bfacda8fd77c81c53fbc869d855050afbd1d69b40a8a6b171a1882b8d19"}
Nov 02 18:22:10.585809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.585748    5042 scope.go:122] "RemoveContainer" containerID="6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd"
Nov 02 18:22:10.600920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.600895    5042 scope.go:122] "RemoveContainer" containerID="6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd"
Nov 02 18:22:10.601276 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:22:10.601255    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd\": container with ID starting with 6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd not found: ID does not exist" containerID="6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd"
Nov 02 18:22:10.601365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.601290    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd"} err="failed to get container status \"6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd\": rpc error: code = NotFound desc = could not find container \"6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd\": container with ID starting with 6b787639981e6aadeed183ed3647e4b340650ef876c5b28e8337d244b8a7aabd not found: ID does not exist"
Nov 02 18:22:10.615613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.615583    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"]
Nov 02 18:22:10.620239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.620212    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-312f21-6984dd475-rgnpn"]
Nov 02 18:22:10.642433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.642398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4hj\" (UniqueName: \"kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.642613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.642462    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.642743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.642721    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.643325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.643301    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.645421 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.645393    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.656299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.656270    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4hj\" (UniqueName: \"kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj\") pod \"maas-api-revoke-sub-afdca042-qvvlq\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:10.705105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:10.705004    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:11.084618 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:22:11.084586    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd666c202_2d1a_4e38_9d50_414071b52ef1.slice/crio-11da406d69a0af745d52f550d58d027259db0bed6d0e1d031a2c1178b77daf27 WatchSource:0}: Error finding container 11da406d69a0af745d52f550d58d027259db0bed6d0e1d031a2c1178b77daf27: Status 404 returned error can't find the container with id 11da406d69a0af745d52f550d58d027259db0bed6d0e1d031a2c1178b77daf27
Nov 02 18:22:11.086906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:11.086875    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"]
Nov 02 18:22:11.611746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:11.611711    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd745b23-0598-4e19-9537-3ed5c422891e" path="/var/lib/kubelet/pods/dd745b23-0598-4e19-9537-3ed5c422891e/volumes"
Nov 02 18:22:11.613855 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:11.613804    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq" event={"ID":"d666c202-2d1a-4e38-9d50-414071b52ef1","Type":"ContainerStarted","Data":"ab0f2a76c4ab13795bf5acf06e3fdd771880b89d1eee587c13080ceec6a12085"}
Nov 02 18:22:11.614001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:11.613921    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq" event={"ID":"d666c202-2d1a-4e38-9d50-414071b52ef1","Type":"ContainerStarted","Data":"11da406d69a0af745d52f550d58d027259db0bed6d0e1d031a2c1178b77daf27"}
Nov 02 18:22:11.635004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:11.634955    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq" podStartSLOduration=1.634941245 podStartE2EDuration="1.634941245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:11.630414492 +0000 UTC m=+2944.209823878" watchObservedRunningTime="2026-11-02 18:22:11.634941245 +0000 UTC m=+2944.214350631"
Nov 02 18:22:12.629929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:12.629889    5042 generic.go:356] "Generic (PLEG): container finished" podID="d666c202-2d1a-4e38-9d50-414071b52ef1" containerID="ab0f2a76c4ab13795bf5acf06e3fdd771880b89d1eee587c13080ceec6a12085" exitCode=0
Nov 02 18:22:12.630516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:12.630489    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq" event={"ID":"d666c202-2d1a-4e38-9d50-414071b52ef1","Type":"ContainerDied","Data":"ab0f2a76c4ab13795bf5acf06e3fdd771880b89d1eee587c13080ceec6a12085"}
Nov 02 18:22:13.832235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.832199    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:13.947925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.942359    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj\" (UniqueName: \"kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj\") pod \"d666c202-2d1a-4e38-9d50-414071b52ef1\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") "
Nov 02 18:22:13.947925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.942723    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca\") pod \"d666c202-2d1a-4e38-9d50-414071b52ef1\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") "
Nov 02 18:22:13.947925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.942816    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca\") pod \"d666c202-2d1a-4e38-9d50-414071b52ef1\" (UID: \"d666c202-2d1a-4e38-9d50-414071b52ef1\") "
Nov 02 18:22:13.985519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.981140    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca" pod "d666c202-2d1a-4e38-9d50-414071b52ef1" (UID: "d666c202-2d1a-4e38-9d50-414071b52ef1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:22:13.991514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.991471    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca" pod "d666c202-2d1a-4e38-9d50-414071b52ef1" (UID: "d666c202-2d1a-4e38-9d50-414071b52ef1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:22:13.997000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:13.996951    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj" pod "d666c202-2d1a-4e38-9d50-414071b52ef1" (UID: "d666c202-2d1a-4e38-9d50-414071b52ef1"). InnerVolumeSpecName "kube-api-access-cj4hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:22:14.057330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:14.057294    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj4hj\" (UniqueName: \"kubernetes.io/projected/d666c202-2d1a-4e38-9d50-414071b52ef1-kube-api-access-cj4hj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:14.057330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:14.057326    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:14.057330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:14.057338    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d666c202-2d1a-4e38-9d50-414071b52ef1-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:14.703525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:14.703488    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq" event={"ID":"d666c202-2d1a-4e38-9d50-414071b52ef1","Type":"ContainerDied","Data":"11da406d69a0af745d52f550d58d027259db0bed6d0e1d031a2c1178b77daf27"}
Nov 02 18:22:14.703742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:14.703543    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11da406d69a0af745d52f550d58d027259db0bed6d0e1d031a2c1178b77daf27"
Nov 02 18:22:14.703742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:14.703608    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"
Nov 02 18:22:18.541330 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:22:18.541289    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:22:20.199215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:20.199167    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:22:21.512377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.512339    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"]
Nov 02 18:22:21.513567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.513543    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d666c202-2d1a-4e38-9d50-414071b52ef1" containerName="revoke-keys"
Nov 02 18:22:21.513567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.513568    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d666c202-2d1a-4e38-9d50-414071b52ef1" containerName="revoke-keys"
Nov 02 18:22:21.513815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.513803    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd705464-df0f-4d2f-bb88-3f9ef9057ad6" containerName="istio-proxy"
Nov 02 18:22:21.513878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.513824    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d666c202-2d1a-4e38-9d50-414071b52ef1" containerName="revoke-keys"
Nov 02 18:22:21.518718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.518698    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.526129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.526102    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:22:21.527685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.527658    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:22:21.534940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.534911    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"]
Nov 02 18:22:21.643597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.643556    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjkm\" (UniqueName: \"kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.643788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.643730    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.643877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.643860    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.746341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.746299    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjkm\" (UniqueName: \"kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.767920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.758239    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.767920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.758398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.769926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.769322    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.770091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.769962    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.784967 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.784551    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjkm\" (UniqueName: \"kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm\") pod \"maas-api-revoke-sub-f41104cb-29r2k\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:21.839912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:21.838042    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:22.141378 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:22:22.140690    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414a1b16_0d3c_4ae8_8b83_26cafc1997b1.slice/crio-4d793bbe74b3542b40877d870c42c85953001d987655eb6e4a98997c44f03263 WatchSource:0}: Error finding container 4d793bbe74b3542b40877d870c42c85953001d987655eb6e4a98997c44f03263: Status 404 returned error can't find the container with id 4d793bbe74b3542b40877d870c42c85953001d987655eb6e4a98997c44f03263
Nov 02 18:22:22.141378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:22.141042    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"]
Nov 02 18:22:23.019865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:23.018980    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k" event={"ID":"414a1b16-0d3c-4ae8-8b83-26cafc1997b1","Type":"ContainerStarted","Data":"ebdcbd979d9bdb137f1d44a4d64731a0eee547710173ef84a9a22ae8f0bcc0e7"}
Nov 02 18:22:23.019865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:23.019052    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k" event={"ID":"414a1b16-0d3c-4ae8-8b83-26cafc1997b1","Type":"ContainerStarted","Data":"4d793bbe74b3542b40877d870c42c85953001d987655eb6e4a98997c44f03263"}
Nov 02 18:22:23.115010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:23.114176    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k" podStartSLOduration=2.114155362 podStartE2EDuration="2.114155362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:23.103579312 +0000 UTC m=+2955.682988698" watchObservedRunningTime="2026-11-02 18:22:23.114155362 +0000 UTC m=+2955.693564749"
Nov 02 18:22:24.051095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.050972    5042 generic.go:356] "Generic (PLEG): container finished" podID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerID="a94f48dd752e7481b78be34df1fe6772e02e86d4494d2412e85400e7753b4aa5" exitCode=0
Nov 02 18:22:24.051593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.051208    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" event={"ID":"5c97ee6f-f862-41b0-84a7-6c264cdc3e37","Type":"ContainerDied","Data":"a94f48dd752e7481b78be34df1fe6772e02e86d4494d2412e85400e7753b4aa5"}
Nov 02 18:22:24.053862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.053799    5042 generic.go:356] "Generic (PLEG): container finished" podID="414a1b16-0d3c-4ae8-8b83-26cafc1997b1" containerID="ebdcbd979d9bdb137f1d44a4d64731a0eee547710173ef84a9a22ae8f0bcc0e7" exitCode=0
Nov 02 18:22:24.053862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.053856    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k" event={"ID":"414a1b16-0d3c-4ae8-8b83-26cafc1997b1","Type":"ContainerDied","Data":"ebdcbd979d9bdb137f1d44a4d64731a0eee547710173ef84a9a22ae8f0bcc0e7"}
Nov 02 18:22:24.205984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.205953    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:22:24.266808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.266733    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume\") pod \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") "
Nov 02 18:22:24.267080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.266833    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7\" (UniqueName: \"kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7\") pod \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\" (UID: \"5c97ee6f-f862-41b0-84a7-6c264cdc3e37\") "
Nov 02 18:22:24.267288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.267220    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume" pod "5c97ee6f-f862-41b0-84a7-6c264cdc3e37" (UID: "5c97ee6f-f862-41b0-84a7-6c264cdc3e37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:22:24.267671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.267644    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:24.269879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.269800    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7" pod "5c97ee6f-f862-41b0-84a7-6c264cdc3e37" (UID: "5c97ee6f-f862-41b0-84a7-6c264cdc3e37"). InnerVolumeSpecName "kube-api-access-w67c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:22:24.369503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:24.369461    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67c7\" (UniqueName: \"kubernetes.io/projected/5c97ee6f-f862-41b0-84a7-6c264cdc3e37-kube-api-access-w67c7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:25.065025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.064994    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"
Nov 02 18:22:25.065515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.065026    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4" event={"ID":"5c97ee6f-f862-41b0-84a7-6c264cdc3e37","Type":"ContainerDied","Data":"86c8f0cf7a2adadce2588a9ccca5b93774a6ad06f03c5d08112373c5ff537509"}
Nov 02 18:22:25.065515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.065091    5042 scope.go:122] "RemoveContainer" containerID="a94f48dd752e7481b78be34df1fe6772e02e86d4494d2412e85400e7753b4aa5"
Nov 02 18:22:25.098665 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.098627    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"]
Nov 02 18:22:25.106492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.106308    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0b0610-667ff7d47c-hq2z4"]
Nov 02 18:22:25.646220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.646174    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" path="/var/lib/kubelet/pods/5c97ee6f-f862-41b0-84a7-6c264cdc3e37/volumes"
Nov 02 18:22:25.709231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.709198    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:25.794508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.794466    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca\") pod \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") "
Nov 02 18:22:25.794707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.794520    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca\") pod \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") "
Nov 02 18:22:25.794786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.794773    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm\" (UniqueName: \"kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm\") pod \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\" (UID: \"414a1b16-0d3c-4ae8-8b83-26cafc1997b1\") "
Nov 02 18:22:25.795018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.794986    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca" pod "414a1b16-0d3c-4ae8-8b83-26cafc1997b1" (UID: "414a1b16-0d3c-4ae8-8b83-26cafc1997b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:22:25.795803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.795779    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:25.797195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.797168    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca" pod "414a1b16-0d3c-4ae8-8b83-26cafc1997b1" (UID: "414a1b16-0d3c-4ae8-8b83-26cafc1997b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:22:25.797373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.797352    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm" pod "414a1b16-0d3c-4ae8-8b83-26cafc1997b1" (UID: "414a1b16-0d3c-4ae8-8b83-26cafc1997b1"). InnerVolumeSpecName "kube-api-access-ptjkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:22:25.898061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.897971    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjkm\" (UniqueName: \"kubernetes.io/projected/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-kube-api-access-ptjkm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:25.898061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:25.898006    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414a1b16-0d3c-4ae8-8b83-26cafc1997b1-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:26.076195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:26.076164    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"
Nov 02 18:22:26.076195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:26.076177    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k" event={"ID":"414a1b16-0d3c-4ae8-8b83-26cafc1997b1","Type":"ContainerDied","Data":"4d793bbe74b3542b40877d870c42c85953001d987655eb6e4a98997c44f03263"}
Nov 02 18:22:26.076704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:26.076227    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d793bbe74b3542b40877d870c42c85953001d987655eb6e4a98997c44f03263"
Nov 02 18:22:27.090918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.090883    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"]
Nov 02 18:22:27.092517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.092491    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerName="payload-processing"
Nov 02 18:22:27.092517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.092516    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerName="payload-processing"
Nov 02 18:22:27.092738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.092625    5042 cpu_manager.go:414] "containerMap: removing container" podUID="414a1b16-0d3c-4ae8-8b83-26cafc1997b1" containerName="revoke-keys"
Nov 02 18:22:27.092738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.092634    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414a1b16-0d3c-4ae8-8b83-26cafc1997b1" containerName="revoke-keys"
Nov 02 18:22:27.092870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.092818    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="414a1b16-0d3c-4ae8-8b83-26cafc1997b1" containerName="revoke-keys"
Nov 02 18:22:27.092934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.092885    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c97ee6f-f862-41b0-84a7-6c264cdc3e37" containerName="payload-processing"
Nov 02 18:22:27.097613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.097583    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.108859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.105569    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cc0b41-openshift-default-dockercfg-6vnnd\""
Nov 02 18:22:27.118996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.118966    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"]
Nov 02 18:22:27.133972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.133928    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134243    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134343    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134388    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134455    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134646    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134670    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134703    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.134913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.134745    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhbv\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243311    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243429    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243479    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243547    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243747    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243778    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243814    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.243879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhbv\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.244059    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.244696    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.245356    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.255203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.245600    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.281872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.268188    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.281872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.268462    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.316867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.309425    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.316867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.309654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.352915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.350414    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.362938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.356719    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhbv\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv\") pod \"e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") " pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:27.443555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:27.443518    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:28.037273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:28.037238    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"]
Nov 02 18:22:28.039602 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:22:28.039575    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2772b5c_8b7b_4e10_8a1c_4d2f3703c2b1.slice/crio-50e2086ec00e9060523b22bb0a584679463263398228491918f6eb7cd965f3b8 WatchSource:0}: Error finding container 50e2086ec00e9060523b22bb0a584679463263398228491918f6eb7cd965f3b8: Status 404 returned error can't find the container with id 50e2086ec00e9060523b22bb0a584679463263398228491918f6eb7cd965f3b8
Nov 02 18:22:28.042521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:28.042482    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:28.042650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:28.042566    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:28.042650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:28.042615    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:28.110179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:28.110146    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" event={"ID":"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1","Type":"ContainerStarted","Data":"50e2086ec00e9060523b22bb0a584679463263398228491918f6eb7cd965f3b8"}
Nov 02 18:22:29.124264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:29.124221    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" event={"ID":"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1","Type":"ContainerStarted","Data":"a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"}
Nov 02 18:22:29.147941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:29.147870    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podStartSLOduration=2.147851037 podStartE2EDuration="2.147851037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:29.145170097 +0000 UTC m=+2961.724579484" watchObservedRunningTime="2026-11-02 18:22:29.147851037 +0000 UTC m=+2961.727260416"
Nov 02 18:22:29.445753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:29.445720    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:29.452301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:29.452273    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:30.133930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:30.133891    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:30.135085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:30.135058    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:22:31.527209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:31.527169    5042 scope.go:122] "RemoveContainer" containerID="c4127b5cee27b284d04dbf8efca832ec64e2c79dd51caa2f85d4168cc5e3c444"
Nov 02 18:22:31.542383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:31.542361    5042 scope.go:122] "RemoveContainer" containerID="a0891be06fe62eee97af817e939e1fc526b7b9d7bea5223eef89aef7b94b22ed"
Nov 02 18:22:37.468867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:37.468043    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:37.921941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:37.919029    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:37.968861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:37.967987    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:38.152120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.152071    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:38.195692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.195643    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:38.544933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.530997    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:38.544933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.531379    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:38.616865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.548304    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 18:22:38.838923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.832553    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:38.890825 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:38.883100    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:39.023900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.023098    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:22:39.024204 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.024042    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:22:39.141156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.065269    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:39.187911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.180263    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:39.208865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.208662    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:22:39.268868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.243474    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"]
Nov 02 18:22:39.269525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.245931    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 18:22:39.295919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.270327    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:22:39.503090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.503036    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:39.574577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.574529    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:39.797958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.777495    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 18:22:39.798425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.798398    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:22:39.817873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.808866    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:22:39.854648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.854616    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:22:39.923922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:39.922234    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:40.086666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.086561    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:40.131920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.131513    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:22:40.131920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.131599    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:22:40.131920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.131641    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk"
Nov 02 18:22:40.131920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.131673    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:22:40.131920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.131706    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:22:40.131920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.131740    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:22:40.132911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.132886    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:22:40.533749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.533713    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"]
Nov 02 18:22:40.542014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.541405    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.559287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.559103    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"]
Nov 02 18:22:40.629234 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.629194    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.629981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.630180    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630325 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.630308    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm645\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630461 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.630440    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.630637    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.630774    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.630949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.630933    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.631136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.631120    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.732852 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.732785    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.732886    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.732970    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733277    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733498 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733441    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733498 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733454    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733511    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733547    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm645\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733587    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733621    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.733755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.733742    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.735753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.734534    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.735753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.735702    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.735997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.735785    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.738019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.737994    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.738133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.738046    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.744166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.744143    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.744495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.744470    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm645\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645\") pod \"maas-default-gateway-openshift-default-65c448b547-b6kzh\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:40.917760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:40.917654    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:41.375270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.374418    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"]
Nov 02 18:22:41.407422 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:22:41.407377    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ffc453_84fb_451d_892b_397ec410a2a1.slice/crio-b3183866081ed4a07c95e2994e9d18251b32448061dede9e5a5383363dea9286 WatchSource:0}: Error finding container b3183866081ed4a07c95e2994e9d18251b32448061dede9e5a5383363dea9286: Status 404 returned error can't find the container with id b3183866081ed4a07c95e2994e9d18251b32448061dede9e5a5383363dea9286
Nov 02 18:22:41.412545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.411403    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:41.412545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.411499    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:41.412545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.411549    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:41.654169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.654075    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" event={"ID":"b1ffc453-84fb-451d-892b-397ec410a2a1","Type":"ContainerStarted","Data":"c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf"}
Nov 02 18:22:41.654169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.654138    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" event={"ID":"b1ffc453-84fb-451d-892b-397ec410a2a1","Type":"ContainerStarted","Data":"b3183866081ed4a07c95e2994e9d18251b32448061dede9e5a5383363dea9286"}
Nov 02 18:22:41.926424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:41.926332    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:42.920368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:42.920310    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:42.920996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:42.920384    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:43.918865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:43.918815    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:43.919097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:43.918892    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:44.269639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.269586    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" podUID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" containerName="payload-processing" containerID="cri-o://6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e" gracePeriod=25
Nov 02 18:22:44.735013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.734978    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:22:44.742051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.742013    5042 generic.go:356] "Generic (PLEG): container finished" podID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" containerID="6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e" exitCode=0
Nov 02 18:22:44.742220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.742099    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"
Nov 02 18:22:44.742287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.742268    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" event={"ID":"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff","Type":"ContainerDied","Data":"6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e"}
Nov 02 18:22:44.742355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.742301    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf" event={"ID":"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff","Type":"ContainerDied","Data":"cf34e9bb9748705500dc4606d7a09e72f86588accfa55b747171a5407a6cd124"}
Nov 02 18:22:44.742355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.742324    5042 scope.go:122] "RemoveContainer" containerID="6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e"
Nov 02 18:22:44.775866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.756775    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podStartSLOduration=4.756756094 podStartE2EDuration="4.756756094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:41.716323609 +0000 UTC m=+2974.295733000" watchObservedRunningTime="2026-11-02 18:22:44.756756094 +0000 UTC m=+2977.336165480"
Nov 02 18:22:44.936720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.936600    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9\" (UniqueName: \"kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9\") pod \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") "
Nov 02 18:22:44.936938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.936882    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume\") pod \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\" (UID: \"f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff\") "
Nov 02 18:22:44.937920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.937894    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume" pod "f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" (UID: "f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:22:44.940931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.939392    5042 scope.go:122] "RemoveContainer" containerID="6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e"
Nov 02 18:22:44.940931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.939797    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:44.940931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.939879    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:44.940931 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:22:44.940432    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e\": container with ID starting with 6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e not found: ID does not exist" containerID="6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e"
Nov 02 18:22:44.940931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.940467    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e"} err="failed to get container status \"6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e\": rpc error: code = NotFound desc = could not find container \"6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e\": container with ID starting with 6775fe0dd232388ccccacf1a23e0fe4cc0d32250dcb760695e756736be52d98e not found: ID does not exist"
Nov 02 18:22:44.941749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:44.941723    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9" pod "f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" (UID: "f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff"). InnerVolumeSpecName "kube-api-access-2dnx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:22:45.039555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.039518    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:45.039555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.039551    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dnx9\" (UniqueName: \"kubernetes.io/projected/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff-kube-api-access-2dnx9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:22:45.070708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.070673    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"]
Nov 02 18:22:45.115196 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.115151    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-22dfc1-6dccdddb7c-bhgtf"]
Nov 02 18:22:45.624315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.624276    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" path="/var/lib/kubelet/pods/f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff/volumes"
Nov 02 18:22:45.918403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.918311    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:45.918403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:45.918380    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:46.934955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:46.932968    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:46.934955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:46.933058    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:47.918480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:47.918444    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:47.920912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:47.918504    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:48.542653 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:22:48.542613    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:22:48.923801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:48.918231    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:48.923801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:48.918332    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:49.918565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:49.918521    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-b6kzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 18:22:49.919145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:49.918594    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 18:22:50.299867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:50.286058    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:22:50.299867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:50.286150    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:51.021041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.020994    5042 generic.go:356] "Generic (PLEG): container finished" podID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerID="e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8" exitCode=0
Nov 02 18:22:51.021722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.021626    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" event={"ID":"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83","Type":"ContainerDied","Data":"e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"}
Nov 02 18:22:51.022380 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.022360    5042 scope.go:122] "RemoveContainer" containerID="e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"
Nov 02 18:22:51.026636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.026594    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:51.026995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.026953    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:51.027315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.027290    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:51.508433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.508270    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:51.510011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.509987    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:51.510862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:51.510826    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:22:52.032342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:52.032309    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" event={"ID":"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83","Type":"ContainerStarted","Data":"403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132"}
Nov 02 18:22:52.559864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:52.559797    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:22:52.563907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:52.561958    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 18:22:52.563907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:52.562029    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 18:22:53.565010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:53.563247    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 18:22:53.565010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:53.563308    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 18:22:54.558547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:54.558515    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 18:22:54.558820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:54.558795    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 18:22:55.558586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.558552    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 18:22:55.559065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.558621    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 18:22:55.722866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.722349    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"]
Nov 02 18:22:55.726392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.723730    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" containerName="payload-processing"
Nov 02 18:22:55.726392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.723755    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" containerName="payload-processing"
Nov 02 18:22:55.726392 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.723961    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4e1042b-e2f4-4de5-b5c3-1ec6c9d497ff" containerName="payload-processing"
Nov 02 18:22:55.735339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.735314    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.735758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.735732    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"]
Nov 02 18:22:55.740259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.740235    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cc0b41-serving-cert\""
Nov 02 18:22:55.750917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.750885    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.751100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.750937    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfcj2\" (UniqueName: \"kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.751100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.751026    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.751331 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.751220    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.941184    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.941254    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfcj2\" (UniqueName: \"kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.941359    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.941556    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:55.942500    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:22:55.959812    5042 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-serving-cert: secret "maas-api-e2e-inf-b-cc0b41-serving-cert" not found
Nov 02 18:22:55.987453 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:22:55.959929    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls podName:530814f9-6689-4091-a006-a7f973b1395f nodeName:}" failed. No retries permitted until 2026-11-02 18:22:56.459902276 +0000 UTC m=+2989.039311654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls") pod "maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" (UID: "530814f9-6689-4091-a006-a7f973b1395f") : secret "maas-api-e2e-inf-b-cc0b41-serving-cert" not found
Nov 02 18:22:56.017336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.017264    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfcj2\" (UniqueName: \"kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:56.027889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.027400    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:56.557390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.557326    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:56.559162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.558677    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 18:22:56.559162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.558744    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 18:22:56.601544 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.601505    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:56.665601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:56.662155    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:57.064307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:57.064277    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"]
Nov 02 18:22:57.065131 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:22:57.065103    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530814f9_6689_4091_a006_a7f973b1395f.slice/crio-2c6f68195a06b3e64c4cfdbdd7ac647b1cfe0bd29929a8c3a69a0516e48f802f WatchSource:0}: Error finding container 2c6f68195a06b3e64c4cfdbdd7ac647b1cfe0bd29929a8c3a69a0516e48f802f: Status 404 returned error can't find the container with id 2c6f68195a06b3e64c4cfdbdd7ac647b1cfe0bd29929a8c3a69a0516e48f802f
Nov 02 18:22:57.066724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:57.066705    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:22:57.118088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:57.117912    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" event={"ID":"530814f9-6689-4091-a006-a7f973b1395f","Type":"ContainerStarted","Data":"2c6f68195a06b3e64c4cfdbdd7ac647b1cfe0bd29929a8c3a69a0516e48f802f"}
Nov 02 18:22:57.559347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:57.559307    5042 patch_prober.go:29] interesting pod/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 18:22:57.560030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:57.559994    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 18:22:58.130429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:58.130319    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" event={"ID":"530814f9-6689-4091-a006-a7f973b1395f","Type":"ContainerStarted","Data":"ce2acf928141597129502884770cf6b75064c9dff34d01a098506d0e807a16b2"}
Nov 02 18:22:58.130429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:58.130417    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:22:58.152070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:58.151942    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" podStartSLOduration=2.558738797 podStartE2EDuration="3.151925626s" totalImagesPullingTime="593.186829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:58.151672584 +0000 UTC m=+2990.731081970" watchObservedRunningTime="2026-11-02 18:22:58.151925626 +0000 UTC m=+2990.731335013"
Nov 02 18:22:58.562939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:58.562902    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:22:58.563404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:58.563385    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:22:58.564038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:22:58.564019    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:23:04.281663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:04.281627    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:23:09.562909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:09.562132    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"]
Nov 02 18:23:09.574014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:09.573981    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"]
Nov 02 18:23:09.729659 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:09.729634    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:23:09.731356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:09.731321    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:23:09.733598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:09.733575    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:23:09.734339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:09.734320    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:23:14.563100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:14.563033    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk" podUID="256a31ec-4dd1-454e-b72b-474fcbf1a835" containerName="payload-processing" containerID="cri-o://7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29" gracePeriod=25
Nov 02 18:23:14.574814 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:14.574753    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerName="payload-pre-processing" containerID="cri-o://1d5b2d2dd57d0358a663d98b98bba5bccc6192f7c3b55eb737bdee6ab9392514" gracePeriod=25
Nov 02 18:23:14.897758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:14.897729    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:23:15.001807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.000276    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume\") pod \"256a31ec-4dd1-454e-b72b-474fcbf1a835\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") "
Nov 02 18:23:15.001807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.000366    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv\" (UniqueName: \"kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv\") pod \"256a31ec-4dd1-454e-b72b-474fcbf1a835\" (UID: \"256a31ec-4dd1-454e-b72b-474fcbf1a835\") "
Nov 02 18:23:15.002629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.002601    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume" pod "256a31ec-4dd1-454e-b72b-474fcbf1a835" (UID: "256a31ec-4dd1-454e-b72b-474fcbf1a835"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:15.005543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.005514    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv" pod "256a31ec-4dd1-454e-b72b-474fcbf1a835" (UID: "256a31ec-4dd1-454e-b72b-474fcbf1a835"). InnerVolumeSpecName "kube-api-access-krbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:15.102715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.102676    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256a31ec-4dd1-454e-b72b-474fcbf1a835-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:15.102715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.102709    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbrv\" (UniqueName: \"kubernetes.io/projected/256a31ec-4dd1-454e-b72b-474fcbf1a835-kube-api-access-krbrv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:15.442350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.442308    5042 generic.go:356] "Generic (PLEG): container finished" podID="256a31ec-4dd1-454e-b72b-474fcbf1a835" containerID="7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29" exitCode=0
Nov 02 18:23:15.442350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.442349    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"
Nov 02 18:23:15.442652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.442393    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk" event={"ID":"256a31ec-4dd1-454e-b72b-474fcbf1a835","Type":"ContainerDied","Data":"7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29"}
Nov 02 18:23:15.442652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.442456    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk" event={"ID":"256a31ec-4dd1-454e-b72b-474fcbf1a835","Type":"ContainerDied","Data":"925aaded61b37d17a0c7b0b8c3ca1ddc4972754deae73be203d8562d91b149d0"}
Nov 02 18:23:15.442652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.442487    5042 scope.go:122] "RemoveContainer" containerID="7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29"
Nov 02 18:23:15.455935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.455917    5042 scope.go:122] "RemoveContainer" containerID="7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29"
Nov 02 18:23:15.456236 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:15.456216    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29\": container with ID starting with 7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29 not found: ID does not exist" containerID="7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29"
Nov 02 18:23:15.456309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.456244    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29"} err="failed to get container status \"7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29\": rpc error: code = NotFound desc = could not find container \"7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29\": container with ID starting with 7c4e06a296016e442fd7eeae5bc5c1b6d65a328553f46ff7b9e2697912a88f29 not found: ID does not exist"
Nov 02 18:23:15.470134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.470102    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"]
Nov 02 18:23:15.474885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.474829    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63be7a-d66cc7bff-s8ggk"]
Nov 02 18:23:15.548010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:15.547972    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256a31ec-4dd1-454e-b72b-474fcbf1a835" path="/var/lib/kubelet/pods/256a31ec-4dd1-454e-b72b-474fcbf1a835/volumes"
Nov 02 18:23:16.137660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:16.137614    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:23:18.541129 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:18.541051    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:23:22.758481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:22.758442    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"]
Nov 02 18:23:22.759136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:22.758639    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy" containerID="cri-o://2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c" gracePeriod=30
Nov 02 18:23:24.874125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.874080    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"]
Nov 02 18:23:24.878014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.875031    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy" containerID="cri-o://6d0261cbdf93548b7254502b814755e9e0dfa2104e9066291c1c4e114fdf5364" gracePeriod=30
Nov 02 18:23:24.880202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.880174    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"]
Nov 02 18:23:24.880656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.880622    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" containerID="cri-o://9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b" gracePeriod=30
Nov 02 18:23:24.893574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.892818    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"]
Nov 02 18:23:24.893574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.893139    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy" containerID="cri-o://497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f" gracePeriod=30
Nov 02 18:23:24.911469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.911428    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"]
Nov 02 18:23:24.912024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:24.911976    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" containerID="cri-o://22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a" gracePeriod=30
Nov 02 18:23:26.138623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:26.138577    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:23:28.319287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.318169    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:23:28.466781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.466595    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.467658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.467625    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.467915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.467892    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.468005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.467967    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.468362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.468334    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:28.468727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.468705    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.469262 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.469236    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:28.469371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.469329    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.471038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.471005    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:28.471148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.471076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.471288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.471269    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.471592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.471546    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:28.511012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.471629    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket\") pod \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\" (UID: \"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8\") "
Nov 02 18:23:28.512148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.512115    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.512298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.512152    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.512298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.512169    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.512298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.512186    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.513114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.512312    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:28.518433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.518369    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:28.518433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.518389    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "kube-api-access-pq7kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:28.519802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.519763    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:23:28.521670 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.521642    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy" pod "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" (UID: "94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:28.614314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.614271    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq7kc\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-kube-api-access-pq7kc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.614314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.614314    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.614557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.614327    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.614557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.614337    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.614557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.614346    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:28.736452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.736414    5042 generic.go:356] "Generic (PLEG): container finished" podID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerID="2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c" exitCode=0
Nov 02 18:23:28.736648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.736477    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"
Nov 02 18:23:28.736648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.736495    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" event={"ID":"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8","Type":"ContainerDied","Data":"2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c"}
Nov 02 18:23:28.736648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.736561    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm" event={"ID":"94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8","Type":"ContainerDied","Data":"d0e782e46aec15fdfc314a8bb973aa73d24fa54a06171c983f713b487dff343c"}
Nov 02 18:23:28.736648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.736597    5042 scope.go:122] "RemoveContainer" containerID="2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c"
Nov 02 18:23:28.754933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.754904    5042 scope.go:122] "RemoveContainer" containerID="2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c"
Nov 02 18:23:28.755279 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:28.755261    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c\": container with ID starting with 2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c not found: ID does not exist" containerID="2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c"
Nov 02 18:23:28.755354 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.755290    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c"} err="failed to get container status \"2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c\": rpc error: code = NotFound desc = could not find container \"2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c\": container with ID starting with 2c3c0c6ede2377c28865739cda86c2c06d4807c677d4d2f63b306e496f89f98c not found: ID does not exist"
Nov 02 18:23:28.795868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.791892    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"]
Nov 02 18:23:28.806601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:28.806563    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-26mbm"]
Nov 02 18:23:29.103153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:29.103049    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:23:29.103452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:29.103414    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:23:29.546905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:29.546834    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" path="/var/lib/kubelet/pods/94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8/volumes"
Nov 02 18:23:29.607509 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:29.607473    5042 patch_prober.go:29] interesting pod/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:23:29.607716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:29.607530    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:23:30.416865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.415237    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:23:30.439620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.439583    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:23:30.569401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569343    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569486    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569559    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569598    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569690    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569720    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569803    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569833    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.569936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569883    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.569953    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570007    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570055    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570083    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570096    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570152    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570233    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570260    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570320    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert\") pod \"19ba2ee9-18e8-4712-ad8b-da7052994a39\" (UID: \"19ba2ee9-18e8-4712-ad8b-da7052994a39\") "
Nov 02 18:23:30.570485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.570378    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t\") pod \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\" (UID: \"b42a4cf5-0cc5-4491-9b14-e1b02cd96498\") "
Nov 02 18:23:30.572217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.571420    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.597033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.572535    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.597033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.572721    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.597033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.580268    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:23:30.597033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.581807    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.597033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.593941    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.616293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.613972    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.616293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.614241    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.619095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.618230    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.619095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.618419    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.622107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.621043    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:30.631975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.630083    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:23:30.631975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.630231    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "kube-api-access-jq8nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:30.631975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.630268    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:23:30.642286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.640551    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:30.642286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.640573    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.642286 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.640558    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:30.642619 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.642359    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t" pod "b42a4cf5-0cc5-4491-9b14-e1b02cd96498" (UID: "b42a4cf5-0cc5-4491-9b14-e1b02cd96498"). InnerVolumeSpecName "kube-api-access-cvh6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:30.642619 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.642448    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy" pod "19ba2ee9-18e8-4712-ad8b-da7052994a39" (UID: "19ba2ee9-18e8-4712-ad8b-da7052994a39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.704866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.701754    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.704866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.701814    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730142    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730282    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730314    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730391    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730607    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730683    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.730746    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy\") pod \"ad488816-a94d-4505-b09b-77abcee964e0\" (UID: \"ad488816-a94d-4505-b09b-77abcee964e0\") "
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732013    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732042    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732055    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732073    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8nj\" (UniqueName: \"kubernetes.io/projected/19ba2ee9-18e8-4712-ad8b-da7052994a39-kube-api-access-jq8nj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732087    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732099    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732113    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732126    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732138    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732150    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732171    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732185    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732203    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ba2ee9-18e8-4712-ad8b-da7052994a39-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732217    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732230    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732244    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ba2ee9-18e8-4712-ad8b-da7052994a39-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.732257    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvh6t\" (UniqueName: \"kubernetes.io/projected/b42a4cf5-0cc5-4491-9b14-e1b02cd96498-kube-api-access-cvh6t\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.748407    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.757074    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.768967    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.783184    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.812798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.783965    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.824924    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.825218    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "kube-api-access-qp5s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.835787    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad488816-a94d-4505-b09b-77abcee964e0-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.835824    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5s6\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-kube-api-access-qp5s6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.835938    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad488816-a94d-4505-b09b-77abcee964e0-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.835956    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.835970    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.835984    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.836016    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.837977    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:30.852721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.840782    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy" pod "ad488816-a94d-4505-b09b-77abcee964e0" (UID: "ad488816-a94d-4505-b09b-77abcee964e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:30.858791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.858710    5042 generic.go:356] "Generic (PLEG): container finished" podID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerID="497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f" exitCode=0
Nov 02 18:23:30.859342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.859220    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" event={"ID":"19ba2ee9-18e8-4712-ad8b-da7052994a39","Type":"ContainerDied","Data":"497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f"}
Nov 02 18:23:30.859989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.859887    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"
Nov 02 18:23:30.860205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.859316    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj" event={"ID":"19ba2ee9-18e8-4712-ad8b-da7052994a39","Type":"ContainerDied","Data":"e0fd78d2350a035dc7f26d0e027a96665a5d1ccfeb52c6c4983038e9d6c47d29"}
Nov 02 18:23:30.860449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.860398    5042 scope.go:122] "RemoveContainer" containerID="497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f"
Nov 02 18:23:30.867774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.867750    5042 generic.go:356] "Generic (PLEG): container finished" podID="ad488816-a94d-4505-b09b-77abcee964e0" containerID="22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a" exitCode=0
Nov 02 18:23:30.868100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.868062    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" event={"ID":"ad488816-a94d-4505-b09b-77abcee964e0","Type":"ContainerDied","Data":"22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a"}
Nov 02 18:23:30.868275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.868257    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg" event={"ID":"ad488816-a94d-4505-b09b-77abcee964e0","Type":"ContainerDied","Data":"07deff1039bf6207e6a7bff8b741cd460a4c32d0142ba82a075a23475524b4c0"}
Nov 02 18:23:30.868511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.868494    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"
Nov 02 18:23:30.892915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.876996    5042 generic.go:356] "Generic (PLEG): container finished" podID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerID="6d0261cbdf93548b7254502b814755e9e0dfa2104e9066291c1c4e114fdf5364" exitCode=0
Nov 02 18:23:30.892915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.877162    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" event={"ID":"b5f2101f-bb0b-4e97-aee9-263faee9ad91","Type":"ContainerDied","Data":"6d0261cbdf93548b7254502b814755e9e0dfa2104e9066291c1c4e114fdf5364"}
Nov 02 18:23:30.892915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.879235    5042 generic.go:356] "Generic (PLEG): container finished" podID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerID="9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b" exitCode=0
Nov 02 18:23:30.892915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.879327    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" event={"ID":"b42a4cf5-0cc5-4491-9b14-e1b02cd96498","Type":"ContainerDied","Data":"9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b"}
Nov 02 18:23:30.892915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.879368    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp" event={"ID":"b42a4cf5-0cc5-4491-9b14-e1b02cd96498","Type":"ContainerDied","Data":"2d2bb355bd312ea318709296cc82cb201f9d43cc3cb0fa36325af8e4d5fda3d6"}
Nov 02 18:23:30.892915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.879461    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"
Nov 02 18:23:30.929150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.929121    5042 scope.go:122] "RemoveContainer" containerID="9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"
Nov 02 18:23:30.938252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.938221    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad488816-a94d-4505-b09b-77abcee964e0-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.938390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.938257    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad488816-a94d-4505-b09b-77abcee964e0-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:30.969862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.969821    5042 scope.go:122] "RemoveContainer" containerID="497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f"
Nov 02 18:23:30.970206 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:30.970181    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f\": container with ID starting with 497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f not found: ID does not exist" containerID="497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f"
Nov 02 18:23:30.970290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.970216    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f"} err="failed to get container status \"497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f\": rpc error: code = NotFound desc = could not find container \"497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f\": container with ID starting with 497ad02242d4bbb6bf9900d37c3e50d5cd0fd4517948764f17ed53bf29e2032f not found: ID does not exist"
Nov 02 18:23:30.970290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.970234    5042 scope.go:122] "RemoveContainer" containerID="9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"
Nov 02 18:23:30.970554 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:30.970528    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4\": container with ID starting with 9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4 not found: ID does not exist" containerID="9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"
Nov 02 18:23:30.970603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.970569    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4"} err="failed to get container status \"9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4\": rpc error: code = NotFound desc = could not find container \"9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4\": container with ID starting with 9b20b7659c86727b6348f6cc98a257e687cc79ff7ae06f77416eb0346230a8f4 not found: ID does not exist"
Nov 02 18:23:30.970603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.970591    5042 scope.go:122] "RemoveContainer" containerID="22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a"
Nov 02 18:23:30.978496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.975032    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:23:30.997064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:30.997025    5042 scope.go:122] "RemoveContainer" containerID="22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a"
Nov 02 18:23:31.011918 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:31.011860    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a\": container with ID starting with 22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a not found: ID does not exist" containerID="22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a"
Nov 02 18:23:31.011918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.011904    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a"} err="failed to get container status \"22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a\": rpc error: code = NotFound desc = could not find container \"22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a\": container with ID starting with 22296acaaab328a0bf11bfb32c8edbfa27b967c8a7b54c343464077ffc8c617a not found: ID does not exist"
Nov 02 18:23:31.012176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.011929    5042 scope.go:122] "RemoveContainer" containerID="9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b"
Nov 02 18:23:31.016205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.016177    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"]
Nov 02 18:23:31.024229 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.024194    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-vqrkp"]
Nov 02 18:23:31.027060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.026927    5042 scope.go:122] "RemoveContainer" containerID="9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b"
Nov 02 18:23:31.027593 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:31.027540    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b\": container with ID starting with 9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b not found: ID does not exist" containerID="9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b"
Nov 02 18:23:31.027593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.027565    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b"} err="failed to get container status \"9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b\": rpc error: code = NotFound desc = could not find container \"9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b\": container with ID starting with 9222ccad423a7056d94c572ccc025844fb08c258d371cf81b7cba1552981c31b not found: ID does not exist"
Nov 02 18:23:31.031287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.031243    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"]
Nov 02 18:23:31.036066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.036044    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-2vgtj"]
Nov 02 18:23:31.039400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039379    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039518    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039608    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039718    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039749    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039771    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039808    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039876    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.039922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039916    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert\") pod \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\" (UID: \"b5f2101f-bb0b-4e97-aee9-263faee9ad91\") "
Nov 02 18:23:31.040084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.039991    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:31.040145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040062    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:31.040145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040106    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:31.040420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040195    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:31.040484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040426    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:31.040705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040688    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.041309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040707    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.041309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040717    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.041309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040726    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.041309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.040736    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.042080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.042058    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:31.042190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.042173    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:23:31.042681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.042659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:31.042799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.042692    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw" pod "b5f2101f-bb0b-4e97-aee9-263faee9ad91" (UID: "b5f2101f-bb0b-4e97-aee9-263faee9ad91"). InnerVolumeSpecName "kube-api-access-pvpjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:31.071719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.071621    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"]
Nov 02 18:23:31.077977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.077940    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-pm5vg"]
Nov 02 18:23:31.142419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.142378    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvpjw\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-kube-api-access-pvpjw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.142419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.142420    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.142655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.142436    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.142655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.142448    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f2101f-bb0b-4e97-aee9-263faee9ad91-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:31.547531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.547500    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" path="/var/lib/kubelet/pods/19ba2ee9-18e8-4712-ad8b-da7052994a39/volumes"
Nov 02 18:23:31.548396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.548365    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad488816-a94d-4505-b09b-77abcee964e0" path="/var/lib/kubelet/pods/ad488816-a94d-4505-b09b-77abcee964e0/volumes"
Nov 02 18:23:31.548996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.548982    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" path="/var/lib/kubelet/pods/b42a4cf5-0cc5-4491-9b14-e1b02cd96498/volumes"
Nov 02 18:23:31.900167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.900145    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"
Nov 02 18:23:31.900606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.900170    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6" event={"ID":"b5f2101f-bb0b-4e97-aee9-263faee9ad91","Type":"ContainerDied","Data":"2d8bfb24b78a6bcfababae5e0f6a84bc335e70bb4d08dfc3c50f662ac7af97de"}
Nov 02 18:23:31.900606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.900242    5042 scope.go:122] "RemoveContainer" containerID="6d0261cbdf93548b7254502b814755e9e0dfa2104e9066291c1c4e114fdf5364"
Nov 02 18:23:31.926628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.926593    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"]
Nov 02 18:23:31.933726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:31.933696    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-63be7a-openshift-default-867d7d59df-4pnw6"]
Nov 02 18:23:33.548121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:33.548089    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" path="/var/lib/kubelet/pods/b5f2101f-bb0b-4e97-aee9-263faee9ad91/volumes"
Nov 02 18:23:35.000676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.000641    5042 generic.go:356] "Generic (PLEG): container finished" podID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerID="1d5b2d2dd57d0358a663d98b98bba5bccc6192f7c3b55eb737bdee6ab9392514" exitCode=0
Nov 02 18:23:35.001152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.000914    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" event={"ID":"4d78a3b0-6db9-4a35-a271-04a31039461f","Type":"ContainerDied","Data":"1d5b2d2dd57d0358a663d98b98bba5bccc6192f7c3b55eb737bdee6ab9392514"}
Nov 02 18:23:35.001152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.000977    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp" event={"ID":"4d78a3b0-6db9-4a35-a271-04a31039461f","Type":"ContainerDied","Data":"106a82cfa89ec003ff1ff377108bfab89b5cc26f2fb5f3c6babceab8b7a33190"}
Nov 02 18:23:35.001152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.001003    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106a82cfa89ec003ff1ff377108bfab89b5cc26f2fb5f3c6babceab8b7a33190"
Nov 02 18:23:35.039695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.039655    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:23:35.143917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.143533    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t\" (UniqueName: \"kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t\") pod \"4d78a3b0-6db9-4a35-a271-04a31039461f\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") "
Nov 02 18:23:35.143917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.143815    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume\") pod \"4d78a3b0-6db9-4a35-a271-04a31039461f\" (UID: \"4d78a3b0-6db9-4a35-a271-04a31039461f\") "
Nov 02 18:23:35.146820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.144212    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume" pod "4d78a3b0-6db9-4a35-a271-04a31039461f" (UID: "4d78a3b0-6db9-4a35-a271-04a31039461f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:35.146820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.144613    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d78a3b0-6db9-4a35-a271-04a31039461f-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:35.146820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.146748    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t" pod "4d78a3b0-6db9-4a35-a271-04a31039461f" (UID: "4d78a3b0-6db9-4a35-a271-04a31039461f"). InnerVolumeSpecName "kube-api-access-4z94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:35.246555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:35.246511    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z94t\" (UniqueName: \"kubernetes.io/projected/4d78a3b0-6db9-4a35-a271-04a31039461f-kube-api-access-4z94t\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:36.012505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:36.012471    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"
Nov 02 18:23:36.037586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:36.037551    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"]
Nov 02 18:23:36.043885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:36.043177    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63be7a-55c7d654f9-h6xnp"]
Nov 02 18:23:37.545671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:37.545633    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" path="/var/lib/kubelet/pods/4d78a3b0-6db9-4a35-a271-04a31039461f/volumes"
Nov 02 18:23:38.048565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.048533    5042 generic.go:356] "Generic (PLEG): container finished" podID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerID="a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8" exitCode=0
Nov 02 18:23:38.048775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.048608    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" event={"ID":"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1","Type":"ContainerDied","Data":"a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"}
Nov 02 18:23:38.049192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.049173    5042 scope.go:122] "RemoveContainer" containerID="a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"
Nov 02 18:23:38.051413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.051384    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:23:38.051493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.051460    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:23:38.051528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.051499    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:23:38.092530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.092265    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"]
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095620    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerName="payload-pre-processing"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095649    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerName="payload-pre-processing"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095682    5042 cpu_manager.go:414] "containerMap: removing container" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095692    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095706    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095714    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095727    5042 cpu_manager.go:414] "containerMap: removing container" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095734    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095747    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095755    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095794    5042 cpu_manager.go:414] "containerMap: removing container" podUID="256a31ec-4dd1-454e-b72b-474fcbf1a835" containerName="payload-processing"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095801    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256a31ec-4dd1-454e-b72b-474fcbf1a835" containerName="payload-processing"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095823    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095831    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095865    5042 cpu_manager.go:414] "containerMap: removing container" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.095872    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096052    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096062    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad488816-a94d-4505-b09b-77abcee964e0" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096073    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ba18d0-f9b6-4c59-823c-3cda2b2dd7a8" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096085    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f2101f-bb0b-4e97-aee9-263faee9ad91" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096101    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="256a31ec-4dd1-454e-b72b-474fcbf1a835" containerName="payload-processing"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096112    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42a4cf5-0cc5-4491-9b14-e1b02cd96498" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096123    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ba2ee9-18e8-4712-ad8b-da7052994a39" containerName="istio-proxy"
Nov 02 18:23:38.099632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.096137    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d78a3b0-6db9-4a35-a271-04a31039461f" containerName="payload-pre-processing"
Nov 02 18:23:38.102143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.102118    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.124584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.124552    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:23:38.124788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.124625    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:23:38.132259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.131153    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"]
Nov 02 18:23:38.228791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.228296    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.228791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.228362    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zdc\" (UniqueName: \"kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.228791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.228432    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.367922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.365968    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.367922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.366066    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zdc\" (UniqueName: \"kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.367922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.366189    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.367922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.367492    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.408369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.408337    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.419249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.419213    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zdc\" (UniqueName: \"kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc\") pod \"maas-api-revoke-sub-668abeec-sxtpd\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.471427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.471381    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:38.765462 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:38.765432    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"]
Nov 02 18:23:38.765824 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:23:38.765734    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d6878e_1294_4f77_8aa5_f963d13c423c.slice/crio-a686e8ce20f5eaca5c37c6b985bceead91d7c00fc043119a1afa1bf129252c7d WatchSource:0}: Error finding container a686e8ce20f5eaca5c37c6b985bceead91d7c00fc043119a1afa1bf129252c7d: Status 404 returned error can't find the container with id a686e8ce20f5eaca5c37c6b985bceead91d7c00fc043119a1afa1bf129252c7d
Nov 02 18:23:39.073959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:39.071189    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd" event={"ID":"12d6878e-1294-4f77-8aa5-f963d13c423c","Type":"ContainerStarted","Data":"a686e8ce20f5eaca5c37c6b985bceead91d7c00fc043119a1afa1bf129252c7d"}
Nov 02 18:23:39.075910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:39.074424    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" event={"ID":"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1","Type":"ContainerStarted","Data":"b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905"}
Nov 02 18:23:39.445903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:39.445871    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:23:39.446307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:39.446284    5042 patch_prober.go:29] interesting pod/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 18:23:39.446362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:39.446334    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 18:23:40.119119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:40.119079    5042 generic.go:356] "Generic (PLEG): container finished" podID="12d6878e-1294-4f77-8aa5-f963d13c423c" containerID="4102bf49ede5e568329243e2f141cffa5e85a64114b0a1a8c016a0a9b26e7658" exitCode=0
Nov 02 18:23:40.119568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:40.119336    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd" event={"ID":"12d6878e-1294-4f77-8aa5-f963d13c423c","Type":"ContainerDied","Data":"4102bf49ede5e568329243e2f141cffa5e85a64114b0a1a8c016a0a9b26e7658"}
Nov 02 18:23:40.446287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:40.446254    5042 patch_prober.go:29] interesting pod/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 18:23:40.446478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:40.446313    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 18:23:41.445363 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.445320    5042 patch_prober.go:29] interesting pod/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 18:23:41.445985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.445382    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 18:23:41.469523 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.469491    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:41.561497 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.561468    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca\") pod \"12d6878e-1294-4f77-8aa5-f963d13c423c\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") "
Nov 02 18:23:41.561812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.561796    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc\" (UniqueName: \"kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc\") pod \"12d6878e-1294-4f77-8aa5-f963d13c423c\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") "
Nov 02 18:23:41.562261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.562236    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca\") pod \"12d6878e-1294-4f77-8aa5-f963d13c423c\" (UID: \"12d6878e-1294-4f77-8aa5-f963d13c423c\") "
Nov 02 18:23:41.563686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.562612    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca" pod "12d6878e-1294-4f77-8aa5-f963d13c423c" (UID: "12d6878e-1294-4f77-8aa5-f963d13c423c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:41.563686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.563658    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:41.565128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.564660    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca" pod "12d6878e-1294-4f77-8aa5-f963d13c423c" (UID: "12d6878e-1294-4f77-8aa5-f963d13c423c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:23:41.565128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.564794    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc" pod "12d6878e-1294-4f77-8aa5-f963d13c423c" (UID: "12d6878e-1294-4f77-8aa5-f963d13c423c"). InnerVolumeSpecName "kube-api-access-22zdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:41.665511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.665477    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12d6878e-1294-4f77-8aa5-f963d13c423c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:41.665511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:41.665508    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zdc\" (UniqueName: \"kubernetes.io/projected/12d6878e-1294-4f77-8aa5-f963d13c423c-kube-api-access-22zdc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:23:42.137110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:42.137070    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"
Nov 02 18:23:42.137110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:42.137096    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd" event={"ID":"12d6878e-1294-4f77-8aa5-f963d13c423c","Type":"ContainerDied","Data":"a686e8ce20f5eaca5c37c6b985bceead91d7c00fc043119a1afa1bf129252c7d"}
Nov 02 18:23:42.137943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:42.137151    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a686e8ce20f5eaca5c37c6b985bceead91d7c00fc043119a1afa1bf129252c7d"
Nov 02 18:23:42.446258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:42.446219    5042 patch_prober.go:29] interesting pod/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 18:23:42.446663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:42.446277    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 18:23:43.452862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:43.449866    5042 patch_prober.go:29] interesting pod/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 18:23:43.452862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:43.449943    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 18:23:44.445936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:44.445897    5042 patch_prober.go:29] interesting pod/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body=
Nov 02 18:23:44.446149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:44.445959    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused"
Nov 02 18:23:45.134268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:45.134218    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:23:45.452137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:45.452101    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:23:45.453402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:23:45.453367    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:23:48.545483 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:23:48.545443    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:24:05.027159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:05.027110    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"]
Nov 02 18:24:05.032170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:05.032138    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94153825-8q94g"]
Nov 02 18:24:05.545036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:05.544997    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790cc34c-dcde-4ac9-af55-78213fad4065" path="/var/lib/kubelet/pods/790cc34c-dcde-4ac9-af55-78213fad4065/volumes"
Nov 02 18:24:15.032713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:15.032673    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"]
Nov 02 18:24:15.037638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:15.037605    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b01f28-rztbh"]
Nov 02 18:24:15.547111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:15.547066    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ebd3de-013a-4b56-874d-368f71ee9149" path="/var/lib/kubelet/pods/78ebd3de-013a-4b56-874d-368f71ee9149/volumes"
Nov 02 18:24:18.543702 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:24:18.543666    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:24:23.467670 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.466483    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"]
Nov 02 18:24:23.468252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.468047    5042 cpu_manager.go:414] "containerMap: removing container" podUID="12d6878e-1294-4f77-8aa5-f963d13c423c" containerName="revoke-keys"
Nov 02 18:24:23.468252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.468073    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d6878e-1294-4f77-8aa5-f963d13c423c" containerName="revoke-keys"
Nov 02 18:24:23.468385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.468260    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d6878e-1294-4f77-8aa5-f963d13c423c" containerName="revoke-keys"
Nov 02 18:24:23.483861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.478429    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.489875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.487820    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:24:23.489875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.487944    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:24:23.512871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.510674    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"]
Nov 02 18:24:23.617276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.617231    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.617501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.617302    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.617501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.617455    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2vj\" (UniqueName: \"kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.718966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.718829    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.718966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.718945    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.719218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.718995    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2vj\" (UniqueName: \"kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.719744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.719657    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.721664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.721641    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.727895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.727834    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2vj\" (UniqueName: \"kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj\") pod \"maas-api-revoke-sub-dfeef8ff-mjxq9\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:23.828910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:23.828873    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:24.125632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:24.125594    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"]
Nov 02 18:24:24.126519 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:24:24.126491    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ac6dd9_c41b_4bbd_bb7d_8ffc71448bde.slice/crio-fa8a0938945ecd5b0a3cc5efd680e2da101767d60788c184d5446a844d8f8b6a WatchSource:0}: Error finding container fa8a0938945ecd5b0a3cc5efd680e2da101767d60788c184d5446a844d8f8b6a: Status 404 returned error can't find the container with id fa8a0938945ecd5b0a3cc5efd680e2da101767d60788c184d5446a844d8f8b6a
Nov 02 18:24:24.712610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:24.712567    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9" event={"ID":"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde","Type":"ContainerStarted","Data":"5a93f28c488532c07a1309ad021cd8157d6d3c85380f5cb1f46a78e64109665d"}
Nov 02 18:24:24.713173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:24.712635    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9" event={"ID":"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde","Type":"ContainerStarted","Data":"fa8a0938945ecd5b0a3cc5efd680e2da101767d60788c184d5446a844d8f8b6a"}
Nov 02 18:24:24.730786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:24.730733    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9" podStartSLOduration=1.73071752 podStartE2EDuration="1.73071752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:24:24.72994625 +0000 UTC m=+3077.309355638" watchObservedRunningTime="2026-11-02 18:24:24.73071752 +0000 UTC m=+3077.310126906"
Nov 02 18:24:25.724602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:25.724568    5042 generic.go:356] "Generic (PLEG): container finished" podID="58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" containerID="5a93f28c488532c07a1309ad021cd8157d6d3c85380f5cb1f46a78e64109665d" exitCode=0
Nov 02 18:24:25.725067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:25.724651    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9" event={"ID":"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde","Type":"ContainerDied","Data":"5a93f28c488532c07a1309ad021cd8157d6d3c85380f5cb1f46a78e64109665d"}
Nov 02 18:24:26.932948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:26.932921    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:27.100941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.094041    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj\" (UniqueName: \"kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj\") pod \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") "
Nov 02 18:24:27.100941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.094124    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca\") pod \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") "
Nov 02 18:24:27.100941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.094169    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca\") pod \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\" (UID: \"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde\") "
Nov 02 18:24:27.107086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.107050    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca" pod "58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" (UID: "58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:27.137291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.137232    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj" pod "58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" (UID: "58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde"). InnerVolumeSpecName "kube-api-access-jq2vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:27.137479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.137457    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca" pod "58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" (UID: "58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:24:27.197523 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.197485    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq2vj\" (UniqueName: \"kubernetes.io/projected/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-kube-api-access-jq2vj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:24:27.197731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.197533    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:24:27.197731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.197550    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:24:27.811150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.806219    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9" event={"ID":"58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde","Type":"ContainerDied","Data":"fa8a0938945ecd5b0a3cc5efd680e2da101767d60788c184d5446a844d8f8b6a"}
Nov 02 18:24:27.811150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.806290    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa8a0938945ecd5b0a3cc5efd680e2da101767d60788c184d5446a844d8f8b6a"
Nov 02 18:24:27.811150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:27.806365    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"
Nov 02 18:24:31.882866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:31.882271    5042 scope.go:122] "RemoveContainer" containerID="1d5b2d2dd57d0358a663d98b98bba5bccc6192f7c3b55eb737bdee6ab9392514"
Nov 02 18:24:32.195668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:32.195627    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:24:48.540827 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:24:48.540738    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:24:54.332978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:54.332929    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 02 18:24:55.986169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.986130    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"]
Nov 02 18:24:55.987367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.987346    5042 cpu_manager.go:414] "containerMap: removing container" podUID="58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" containerName="revoke-keys"
Nov 02 18:24:55.987442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.987370    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" containerName="revoke-keys"
Nov 02 18:24:55.987557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.987544    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" containerName="revoke-keys"
Nov 02 18:24:55.992829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.992796    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:55.998878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.998832    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:24:55.999058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.998877    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:24:56.000413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:55.999984    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"]
Nov 02 18:24:56.104333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.104297    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.104526 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.104503    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h67lq\" (UniqueName: \"kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.104575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.104541    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.206493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.206458    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.206694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.206607    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h67lq\" (UniqueName: \"kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.206694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.206642    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.207451 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.207423    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.209219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.209191    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.216120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.216064    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h67lq\" (UniqueName: \"kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq\") pod \"maas-api-revoke-sub-4d68f851-w5cdw\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.321066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.320970    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:56.620014 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:24:56.619968    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a404b2b_754c_4c5b_8853_13afe0b0897a.slice/crio-3384a862ebda616bf82fd5a55d26cbe6133a35f8660ecb8fd3339c48591ac682 WatchSource:0}: Error finding container 3384a862ebda616bf82fd5a55d26cbe6133a35f8660ecb8fd3339c48591ac682: Status 404 returned error can't find the container with id 3384a862ebda616bf82fd5a55d26cbe6133a35f8660ecb8fd3339c48591ac682
Nov 02 18:24:56.620524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.620497    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"]
Nov 02 18:24:56.646041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:56.646005    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw" event={"ID":"5a404b2b-754c-4c5b-8853-13afe0b0897a","Type":"ContainerStarted","Data":"3384a862ebda616bf82fd5a55d26cbe6133a35f8660ecb8fd3339c48591ac682"}
Nov 02 18:24:57.704961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:57.699768    5042 generic.go:356] "Generic (PLEG): container finished" podID="5a404b2b-754c-4c5b-8853-13afe0b0897a" containerID="7ecf841c1f053769ff5a59979dbe1738e49545f284b1a11ed7b358633901ad51" exitCode=0
Nov 02 18:24:57.704961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:57.700046    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw" event={"ID":"5a404b2b-754c-4c5b-8853-13afe0b0897a","Type":"ContainerDied","Data":"7ecf841c1f053769ff5a59979dbe1738e49545f284b1a11ed7b358633901ad51"}
Nov 02 18:24:58.898380 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.898343    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:58.979054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.978943    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca\") pod \"5a404b2b-754c-4c5b-8853-13afe0b0897a\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") "
Nov 02 18:24:58.979249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.979133    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca\") pod \"5a404b2b-754c-4c5b-8853-13afe0b0897a\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") "
Nov 02 18:24:58.979249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.979163    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq\" (UniqueName: \"kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq\") pod \"5a404b2b-754c-4c5b-8853-13afe0b0897a\" (UID: \"5a404b2b-754c-4c5b-8853-13afe0b0897a\") "
Nov 02 18:24:58.979386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.979348    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca" pod "5a404b2b-754c-4c5b-8853-13afe0b0897a" (UID: "5a404b2b-754c-4c5b-8853-13afe0b0897a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:24:58.980073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.979974    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:24:58.981695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.981666    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq" pod "5a404b2b-754c-4c5b-8853-13afe0b0897a" (UID: "5a404b2b-754c-4c5b-8853-13afe0b0897a"). InnerVolumeSpecName "kube-api-access-h67lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:24:58.981695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:58.981670    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca" pod "5a404b2b-754c-4c5b-8853-13afe0b0897a" (UID: "5a404b2b-754c-4c5b-8853-13afe0b0897a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:24:59.082029 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:59.081992    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a404b2b-754c-4c5b-8853-13afe0b0897a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:24:59.082029 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:59.082022    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h67lq\" (UniqueName: \"kubernetes.io/projected/5a404b2b-754c-4c5b-8853-13afe0b0897a-kube-api-access-h67lq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:24:59.719008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:59.718980    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"
Nov 02 18:24:59.719224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:59.719012    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw" event={"ID":"5a404b2b-754c-4c5b-8853-13afe0b0897a","Type":"ContainerDied","Data":"3384a862ebda616bf82fd5a55d26cbe6133a35f8660ecb8fd3339c48591ac682"}
Nov 02 18:24:59.719224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:24:59.719064    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3384a862ebda616bf82fd5a55d26cbe6133a35f8660ecb8fd3339c48591ac682"
Nov 02 18:25:06.984004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.983965    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"]
Nov 02 18:25:06.984723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.984706    5042 cpu_manager.go:414] "containerMap: removing container" podUID="5a404b2b-754c-4c5b-8853-13afe0b0897a" containerName="revoke-keys"
Nov 02 18:25:06.984723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.984724    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a404b2b-754c-4c5b-8853-13afe0b0897a" containerName="revoke-keys"
Nov 02 18:25:06.984913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.984902    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a404b2b-754c-4c5b-8853-13afe0b0897a" containerName="revoke-keys"
Nov 02 18:25:06.988775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.988746    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:06.993794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.993767    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:25:06.994735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:06.994686    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:25:07.007022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.000581    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"]
Nov 02 18:25:07.151672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.151617    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.152072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.152041    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.152481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.152460    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq2c\" (UniqueName: \"kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.254762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.254673    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq2c\" (UniqueName: \"kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.254978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.254827    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.254978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.254905    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.255599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.255565    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.257426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.257402    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.263507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.263477    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq2c\" (UniqueName: \"kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c\") pod \"maas-api-revoke-sub-b307226f-jhqw8\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.316167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.316134    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:07.609224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.609191    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"]
Nov 02 18:25:07.609976 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:25:07.609950    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c729eb_e471_489e_b647_77062c55a25c.slice/crio-2805ef9110d70b9cf8cf534f7c54f6b5ed163775c547f9bb95f1fa2209aeeb41 WatchSource:0}: Error finding container 2805ef9110d70b9cf8cf534f7c54f6b5ed163775c547f9bb95f1fa2209aeeb41: Status 404 returned error can't find the container with id 2805ef9110d70b9cf8cf534f7c54f6b5ed163775c547f9bb95f1fa2209aeeb41
Nov 02 18:25:07.822232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:07.822194    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8" event={"ID":"33c729eb-e471-489e-b647-77062c55a25c","Type":"ContainerStarted","Data":"2805ef9110d70b9cf8cf534f7c54f6b5ed163775c547f9bb95f1fa2209aeeb41"}
Nov 02 18:25:08.350618 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.350518    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"]
Nov 02 18:25:08.358146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.358115    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.369602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.369565    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8a098971-openshift-default-dockercfg-jspbn\""
Nov 02 18:25:08.378172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378134    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.378311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378203    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m7tz\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.378401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378347    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.378469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378435    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.378469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378461    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.378591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378496    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.378591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378550    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.379732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378601    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.379732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.378657    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.379912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.379749    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"]
Nov 02 18:25:08.506860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.480001    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.507289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.507255    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.507443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.507427    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.507573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.507559    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.507730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.507715    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.517002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.516961    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.517309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.517288    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.524769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.517529    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.525084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.525054    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m7tz\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.529875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.509036    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.530101 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.509907    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.530212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.510170    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.530315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.509258    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.530395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.530272    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.576931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.575419    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.599864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.586500    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.599864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.587534    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m7tz\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.599864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.588513    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy\") pod \"e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") " pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.699644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.696787    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:08.848916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.848882    5042 generic.go:356] "Generic (PLEG): container finished" podID="33c729eb-e471-489e-b647-77062c55a25c" containerID="b819e0f255bed8c35d4a8521f44668c59ed8300bc91618275a1180933ca052d6" exitCode=0
Nov 02 18:25:08.849341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:08.849307    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8" event={"ID":"33c729eb-e471-489e-b647-77062c55a25c","Type":"ContainerDied","Data":"b819e0f255bed8c35d4a8521f44668c59ed8300bc91618275a1180933ca052d6"}
Nov 02 18:25:09.109374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.109336    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"]
Nov 02 18:25:09.109502 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:25:09.109388    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0f812c_97f1_4a4d_9ee3_9d135fd3b559.slice/crio-07d9088d03ed7362254986e35b7f32513f5cb01ec6e8e240176fb9fbba9697e1 WatchSource:0}: Error finding container 07d9088d03ed7362254986e35b7f32513f5cb01ec6e8e240176fb9fbba9697e1: Status 404 returned error can't find the container with id 07d9088d03ed7362254986e35b7f32513f5cb01ec6e8e240176fb9fbba9697e1
Nov 02 18:25:09.113240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.113205    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:09.113353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.113275    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:09.113353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.113305    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:09.899510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.899474    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" event={"ID":"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559","Type":"ContainerStarted","Data":"1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"}
Nov 02 18:25:09.900076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.899525    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" event={"ID":"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559","Type":"ContainerStarted","Data":"07d9088d03ed7362254986e35b7f32513f5cb01ec6e8e240176fb9fbba9697e1"}
Nov 02 18:25:09.927993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:09.927076    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podStartSLOduration=1.927062222 podStartE2EDuration="1.927062222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:09.923809431 +0000 UTC m=+3122.503218818" watchObservedRunningTime="2026-11-02 18:25:09.927062222 +0000 UTC m=+3122.506471608"
Nov 02 18:25:10.175815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.175747    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:10.238025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.237994    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca\") pod \"33c729eb-e471-489e-b647-77062c55a25c\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") "
Nov 02 18:25:10.238219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.238045    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c\" (UniqueName: \"kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c\") pod \"33c729eb-e471-489e-b647-77062c55a25c\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") "
Nov 02 18:25:10.238314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.238288    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca\") pod \"33c729eb-e471-489e-b647-77062c55a25c\" (UID: \"33c729eb-e471-489e-b647-77062c55a25c\") "
Nov 02 18:25:10.238804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.238771    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca" pod "33c729eb-e471-489e-b647-77062c55a25c" (UID: "33c729eb-e471-489e-b647-77062c55a25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:10.239135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.239030    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c729eb-e471-489e-b647-77062c55a25c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:10.240633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.240610    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca" pod "33c729eb-e471-489e-b647-77062c55a25c" (UID: "33c729eb-e471-489e-b647-77062c55a25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:10.240721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.240667    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c" pod "33c729eb-e471-489e-b647-77062c55a25c" (UID: "33c729eb-e471-489e-b647-77062c55a25c"). InnerVolumeSpecName "kube-api-access-bmq2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:10.340602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.340566    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c729eb-e471-489e-b647-77062c55a25c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:10.340602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.340597    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmq2c\" (UniqueName: \"kubernetes.io/projected/33c729eb-e471-489e-b647-77062c55a25c-kube-api-access-bmq2c\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:10.703868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.698127    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:10.715643 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.715361    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:10.909766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.909732    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"
Nov 02 18:25:10.910287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.909765    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8" event={"ID":"33c729eb-e471-489e-b647-77062c55a25c","Type":"ContainerDied","Data":"2805ef9110d70b9cf8cf534f7c54f6b5ed163775c547f9bb95f1fa2209aeeb41"}
Nov 02 18:25:10.910287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.909811    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2805ef9110d70b9cf8cf534f7c54f6b5ed163775c547f9bb95f1fa2209aeeb41"
Nov 02 18:25:10.910287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.910093    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:10.911139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:10.911102    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:25:12.028942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:12.028905    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"]
Nov 02 18:25:12.036005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:12.035976    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d50fc2-ln9pf"]
Nov 02 18:25:13.325563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.325527    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"]
Nov 02 18:25:13.326374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.326353    5042 cpu_manager.go:414] "containerMap: removing container" podUID="33c729eb-e471-489e-b647-77062c55a25c" containerName="revoke-keys"
Nov 02 18:25:13.326374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.326374    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c729eb-e471-489e-b647-77062c55a25c" containerName="revoke-keys"
Nov 02 18:25:13.326536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.326496    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c729eb-e471-489e-b647-77062c55a25c" containerName="revoke-keys"
Nov 02 18:25:13.330510 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.330491    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.337815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.337788    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8a098971-dockercfg-xw5d6\""
Nov 02 18:25:13.337956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.337857    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8a098971\""
Nov 02 18:25:13.345135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.345106    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"]
Nov 02 18:25:13.351755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.351730    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"]
Nov 02 18:25:13.356789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.356759    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.365675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.365622    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"]
Nov 02 18:25:13.430604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.430574    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.430791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.430719    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-8a098971-569568574f-d27dl\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") " pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.430852 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.430795    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cdj7\" (UniqueName: \"kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7\") pod \"payload-processing-e2e-mt-8a098971-569568574f-d27dl\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") " pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.431024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.431006    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6\") pod \"payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.544207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.544161    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6\") pod \"payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.570924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.547501    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.571491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.571428    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-8a098971-569568574f-d27dl\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") " pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.571740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.571718    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cdj7\" (UniqueName: \"kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7\") pod \"payload-processing-e2e-mt-8a098971-569568574f-d27dl\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") " pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.586296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.574072    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-8a098971-569568574f-d27dl\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") " pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.586541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.577458    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.633505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.633462    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cdj7\" (UniqueName: \"kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7\") pod \"payload-processing-e2e-mt-8a098971-569568574f-d27dl\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") " pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.661904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.659732    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6\") pod \"payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.678868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.671504    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:13.703220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.700351    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:13.817663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:13.815150    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709756ae-aedd-4054-a282-c010a187b4d3" path="/var/lib/kubelet/pods/709756ae-aedd-4054-a282-c010a187b4d3/volumes"
Nov 02 18:25:14.177858 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:25:14.177811    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92574ad_87f5_408e_a4dc_1b1c46bc69a6.slice/crio-5694a25119fdcb93ea862e87eb1d58e8f5db69ccc4b124ccd420a24613653fa8 WatchSource:0}: Error finding container 5694a25119fdcb93ea862e87eb1d58e8f5db69ccc4b124ccd420a24613653fa8: Status 404 returned error can't find the container with id 5694a25119fdcb93ea862e87eb1d58e8f5db69ccc4b124ccd420a24613653fa8
Nov 02 18:25:14.180743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.180713    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"]
Nov 02 18:25:14.197717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.196331    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"]
Nov 02 18:25:14.992221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.992180    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl" event={"ID":"c92574ad-87f5-408e-a4dc-1b1c46bc69a6","Type":"ContainerStarted","Data":"b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d"}
Nov 02 18:25:14.992819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.992794    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl" event={"ID":"c92574ad-87f5-408e-a4dc-1b1c46bc69a6","Type":"ContainerStarted","Data":"5694a25119fdcb93ea862e87eb1d58e8f5db69ccc4b124ccd420a24613653fa8"}
Nov 02 18:25:14.992998 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.992981    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:14.995673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.995646    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd" event={"ID":"ed73717a-e17c-4cb4-8b93-57e43a0bc919","Type":"ContainerStarted","Data":"c58175d164acc78afdc570038688729b1fdc4ab3ae9ad1de7fdf8a5a233030cc"}
Nov 02 18:25:14.995952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.995932    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd" event={"ID":"ed73717a-e17c-4cb4-8b93-57e43a0bc919","Type":"ContainerStarted","Data":"affc70d9d3a9adcf523a014d1e3aa9f5de4f93ce724ee775980e2e0acb9aebf6"}
Nov 02 18:25:14.996114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:14.996101    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:15.013337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:15.013283    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl" podStartSLOduration=2.013267133 podStartE2EDuration="2.013267133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:15.012129463 +0000 UTC m=+3127.591538852" watchObservedRunningTime="2026-11-02 18:25:15.013267133 +0000 UTC m=+3127.592676550"
Nov 02 18:25:15.033613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:15.033547    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd" podStartSLOduration=2.033528754 podStartE2EDuration="2.033528754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:15.028099976 +0000 UTC m=+3127.607509359" watchObservedRunningTime="2026-11-02 18:25:15.033528754 +0000 UTC m=+3127.612938141"
Nov 02 18:25:18.541461 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:25:18.541419    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:25:23.271695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:23.271660    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"]
Nov 02 18:25:24.029804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:24.029765    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"]
Nov 02 18:25:24.034385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:24.034358    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19ad422-5vvgj"]
Nov 02 18:25:25.545580 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:25.545548    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c24635b-b6fd-49d2-a23b-ef78a00b20cd" path="/var/lib/kubelet/pods/1c24635b-b6fd-49d2-a23b-ef78a00b20cd/volumes"
Nov 02 18:25:26.011254 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:26.011218    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:25:26.011480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:26.011315    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:25:28.274918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:28.274798    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerName="payload-pre-processing" containerID="cri-o://18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717" gracePeriod=25
Nov 02 18:25:29.644074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:29.644033    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"]
Nov 02 18:25:29.644539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:29.644301    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2" podUID="7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" containerName="maas-api" containerID="cri-o://b06108e9e184f65301022ffc17346dcd9821974c30243ad845bda08ff66e8e59" gracePeriod=30
Nov 02 18:25:30.288648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.288604    5042 generic.go:356] "Generic (PLEG): container finished" podID="7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" containerID="b06108e9e184f65301022ffc17346dcd9821974c30243ad845bda08ff66e8e59" exitCode=0
Nov 02 18:25:30.288933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.288777    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2" event={"ID":"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af","Type":"ContainerDied","Data":"b06108e9e184f65301022ffc17346dcd9821974c30243ad845bda08ff66e8e59"}
Nov 02 18:25:30.368360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.368331    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:25:30.497044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.496980    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs\") pod \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") "
Nov 02 18:25:30.497339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.497076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj\" (UniqueName: \"kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj\") pod \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") "
Nov 02 18:25:30.497443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.497407    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls\") pod \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") "
Nov 02 18:25:30.497556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.497465    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca\") pod \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\" (UID: \"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af\") "
Nov 02 18:25:30.502064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.502005    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca" pod "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" (UID: "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:30.512297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.512241    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs" pod "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" (UID: "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:30.516959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.516922    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls" pod "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" (UID: "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:30.521820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.521781    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj" pod "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" (UID: "7ec5ab2d-3e3d-4f09-bc06-41e30894c7af"). InnerVolumeSpecName "kube-api-access-vgkdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:30.600073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.600030    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:30.600073 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.600071    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:30.600335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.600091    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkdj\" (UniqueName: \"kubernetes.io/projected/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-kube-api-access-vgkdj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:30.600335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:30.600106    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:31.311715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:31.311674    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2" event={"ID":"7ec5ab2d-3e3d-4f09-bc06-41e30894c7af","Type":"ContainerDied","Data":"3cc601da5a8778a3701b8ed917a79540e0a11cf04ac05078ae4c517f65a115b6"}
Nov 02 18:25:31.312260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:31.311756    5042 scope.go:122] "RemoveContainer" containerID="b06108e9e184f65301022ffc17346dcd9821974c30243ad845bda08ff66e8e59"
Nov 02 18:25:31.312260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:31.311962    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"
Nov 02 18:25:31.349338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:31.349302    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"]
Nov 02 18:25:31.356176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:31.356004    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50b3a6-67948cdbff-kx8g2"]
Nov 02 18:25:31.546108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:31.546076    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" path="/var/lib/kubelet/pods/7ec5ab2d-3e3d-4f09-bc06-41e30894c7af/volumes"
Nov 02 18:25:32.258865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:32.253900    5042 scope.go:122] "RemoveContainer" containerID="e1aa315b2a63d0f31147e85a33e4281b286717436676a9e88be01d7ddba121e0"
Nov 02 18:25:32.394225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:32.393176    5042 scope.go:122] "RemoveContainer" containerID="d826ef2f37d3f207c30cf119bff28114e83691044fd0906d5c3cdd836b8e4dd9"
Nov 02 18:25:34.432860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.432369    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"]
Nov 02 18:25:34.438415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.433554    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" containerName="maas-api"
Nov 02 18:25:34.438415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.433582    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" containerName="maas-api"
Nov 02 18:25:34.438415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.433911    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec5ab2d-3e3d-4f09-bc06-41e30894c7af" containerName="maas-api"
Nov 02 18:25:34.440687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.440356    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.447164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.446223    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8a098971-serving-cert\""
Nov 02 18:25:34.455435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.447970    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"]
Nov 02 18:25:34.455435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.448825    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.455435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.449022    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.455435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.449119    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.455435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.449159    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf6j6\" (UniqueName: \"kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.551146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.551106    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.551344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.551220    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.551344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.551280    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.551344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.551305    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf6j6\" (UniqueName: \"kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.552062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.552037    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.553980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.553961    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.554072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.553993    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.560233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.560209    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf6j6\" (UniqueName: \"kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6\") pod \"maas-api-e2e-mt-8a098971-75499b458c-c2vtg\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.763688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.763654    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:34.884378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:34.884330    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:25:35.165460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:35.165430    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"]
Nov 02 18:25:35.165582 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:25:35.165501    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc6f1d9_87dc_454d_8951_55a5626905a0.slice/crio-4627cc9e9b1e9957cfcb585e0a9cda66db7e98b59ccf536e6080cc68a4034c4d WatchSource:0}: Error finding container 4627cc9e9b1e9957cfcb585e0a9cda66db7e98b59ccf536e6080cc68a4034c4d: Status 404 returned error can't find the container with id 4627cc9e9b1e9957cfcb585e0a9cda66db7e98b59ccf536e6080cc68a4034c4d
Nov 02 18:25:35.518725 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:35.518678    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg" event={"ID":"afc6f1d9-87dc-454d-8951-55a5626905a0","Type":"ContainerStarted","Data":"4627cc9e9b1e9957cfcb585e0a9cda66db7e98b59ccf536e6080cc68a4034c4d"}
Nov 02 18:25:36.529980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:36.529940    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg" event={"ID":"afc6f1d9-87dc-454d-8951-55a5626905a0","Type":"ContainerStarted","Data":"f38cdc01ad48a1849a689ace11b03880b7fa2381426a961c8507e22f3a5b0e75"}
Nov 02 18:25:36.530434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:36.530057    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:36.554117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:36.554064    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg" podStartSLOduration=1.795524539 podStartE2EDuration="2.554049726s" totalImagesPullingTime="758.525187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:36.54923351 +0000 UTC m=+3149.128642899" watchObservedRunningTime="2026-11-02 18:25:36.554049726 +0000 UTC m=+3149.133459112"
Nov 02 18:25:42.546634 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:42.546598    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:25:44.848802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:44.848761    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:25:48.648425 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:25:48.648387    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:25:48.665730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.665703    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:25:48.723314 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.723273    5042 generic.go:356] "Generic (PLEG): container finished" podID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerID="18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717" exitCode=0
Nov 02 18:25:48.723532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.723329    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"
Nov 02 18:25:48.723532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.723351    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" event={"ID":"72ee01e7-6cc0-4218-aa5d-65c64cfff50a","Type":"ContainerDied","Data":"18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717"}
Nov 02 18:25:48.723532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.723409    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s" event={"ID":"72ee01e7-6cc0-4218-aa5d-65c64cfff50a","Type":"ContainerDied","Data":"ab8b7712b88cab9da29a9587d3657cc3bf47eef2482657584abfe5af18591edf"}
Nov 02 18:25:48.723532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.723435    5042 scope.go:122] "RemoveContainer" containerID="18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717"
Nov 02 18:25:48.739948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.739921    5042 scope.go:122] "RemoveContainer" containerID="18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717"
Nov 02 18:25:48.740310 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:25:48.740286    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717\": container with ID starting with 18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717 not found: ID does not exist" containerID="18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717"
Nov 02 18:25:48.740416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.740317    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717"} err="failed to get container status \"18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717\": rpc error: code = NotFound desc = could not find container \"18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717\": container with ID starting with 18e31bfcbe17c14ae27b776e5092b4c08be2a35f61f2fc595efa48142e0e4717 not found: ID does not exist"
Nov 02 18:25:48.804997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.804964    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz\" (UniqueName: \"kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz\") pod \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") "
Nov 02 18:25:48.805180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.805011    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume\") pod \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\" (UID: \"72ee01e7-6cc0-4218-aa5d-65c64cfff50a\") "
Nov 02 18:25:48.805513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.805471    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume" pod "72ee01e7-6cc0-4218-aa5d-65c64cfff50a" (UID: "72ee01e7-6cc0-4218-aa5d-65c64cfff50a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:48.806026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.806006    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:48.807499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.807471    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz" pod "72ee01e7-6cc0-4218-aa5d-65c64cfff50a" (UID: "72ee01e7-6cc0-4218-aa5d-65c64cfff50a"). InnerVolumeSpecName "kube-api-access-kh2cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:48.907341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:48.907233    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2cz\" (UniqueName: \"kubernetes.io/projected/72ee01e7-6cc0-4218-aa5d-65c64cfff50a-kube-api-access-kh2cz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:25:49.049895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:49.049857    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"]
Nov 02 18:25:49.055901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:49.055873    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b3a6-cf79d6cf-w785s"]
Nov 02 18:25:49.546268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:25:49.546229    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" path="/var/lib/kubelet/pods/72ee01e7-6cc0-4218-aa5d-65c64cfff50a/volumes"
Nov 02 18:26:04.349688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.349645    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"]
Nov 02 18:26:04.350759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.350733    5042 cpu_manager.go:414] "containerMap: removing container" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerName="payload-pre-processing"
Nov 02 18:26:04.350927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.350761    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerName="payload-pre-processing"
Nov 02 18:26:04.351028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.351010    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ee01e7-6cc0-4218-aa5d-65c64cfff50a" containerName="payload-pre-processing"
Nov 02 18:26:04.393649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.393610    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"]
Nov 02 18:26:04.393876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.393739    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.398133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.398103    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:26:04.398772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.398751    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:26:04.551270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.551230    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.551476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.551289    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klp2r\" (UniqueName: \"kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.551550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.551496    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.653734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.653620    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.653734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.653673    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klp2r\" (UniqueName: \"kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.654229 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.653886    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.654495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.654464    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.656609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.656580    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.663202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.663175    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klp2r\" (UniqueName: \"kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r\") pod \"maas-api-revoke-sub-5e70e438-jmcww\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:04.738880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:04.738792    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:05.135484 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:26:05.135445    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b075f3_7946_46f3_b9bb_b0b479bafa3c.slice/crio-666a0c5a1bcfb7db47d2d65fb7b091444d5bf4e6bf24808684e69aa85dc2bdfb WatchSource:0}: Error finding container 666a0c5a1bcfb7db47d2d65fb7b091444d5bf4e6bf24808684e69aa85dc2bdfb: Status 404 returned error can't find the container with id 666a0c5a1bcfb7db47d2d65fb7b091444d5bf4e6bf24808684e69aa85dc2bdfb
Nov 02 18:26:05.137435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:05.137351    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"]
Nov 02 18:26:06.057572 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:06.057530    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww" event={"ID":"33b075f3-7946-46f3-b9bb-b0b479bafa3c","Type":"ContainerStarted","Data":"e714ceca5bbad6538e4e050022b18ec5982ad6b4a9f954ed9d067035c6953919"}
Nov 02 18:26:06.058188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:06.057601    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww" event={"ID":"33b075f3-7946-46f3-b9bb-b0b479bafa3c","Type":"ContainerStarted","Data":"666a0c5a1bcfb7db47d2d65fb7b091444d5bf4e6bf24808684e69aa85dc2bdfb"}
Nov 02 18:26:06.078648 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:06.078588    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww" podStartSLOduration=2.078572879 podStartE2EDuration="2.078572879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:06.074956503 +0000 UTC m=+3178.654365890" watchObservedRunningTime="2026-11-02 18:26:06.078572879 +0000 UTC m=+3178.657982265"
Nov 02 18:26:07.102472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:07.102433    5042 generic.go:356] "Generic (PLEG): container finished" podID="33b075f3-7946-46f3-b9bb-b0b479bafa3c" containerID="e714ceca5bbad6538e4e050022b18ec5982ad6b4a9f954ed9d067035c6953919" exitCode=0
Nov 02 18:26:07.102997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:07.102521    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww" event={"ID":"33b075f3-7946-46f3-b9bb-b0b479bafa3c","Type":"ContainerDied","Data":"e714ceca5bbad6538e4e050022b18ec5982ad6b4a9f954ed9d067035c6953919"}
Nov 02 18:26:08.114863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.114809    5042 generic.go:356] "Generic (PLEG): container finished" podID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerID="1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a" exitCode=0
Nov 02 18:26:08.115407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.114885    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" event={"ID":"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559","Type":"ContainerDied","Data":"1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"}
Nov 02 18:26:08.115983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.115550    5042 scope.go:122] "RemoveContainer" containerID="1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"
Nov 02 18:26:08.117852 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.117810    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:08.118000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.117900    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:08.118000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.117933    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:08.405419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.405385    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:08.543999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.543962    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r\" (UniqueName: \"kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r\") pod \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") "
Nov 02 18:26:08.544266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.544248    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca\") pod \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") "
Nov 02 18:26:08.544381 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.544336    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca\") pod \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\" (UID: \"33b075f3-7946-46f3-b9bb-b0b479bafa3c\") "
Nov 02 18:26:08.544979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.544916    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca" pod "33b075f3-7946-46f3-b9bb-b0b479bafa3c" (UID: "33b075f3-7946-46f3-b9bb-b0b479bafa3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:08.546668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.546634    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r" pod "33b075f3-7946-46f3-b9bb-b0b479bafa3c" (UID: "33b075f3-7946-46f3-b9bb-b0b479bafa3c"). InnerVolumeSpecName "kube-api-access-klp2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:08.547113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.547090    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca" pod "33b075f3-7946-46f3-b9bb-b0b479bafa3c" (UID: "33b075f3-7946-46f3-b9bb-b0b479bafa3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:26:08.651984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.651438    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:08.651984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.651479    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33b075f3-7946-46f3-b9bb-b0b479bafa3c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:08.651984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:08.651495    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klp2r\" (UniqueName: \"kubernetes.io/projected/33b075f3-7946-46f3-b9bb-b0b479bafa3c-kube-api-access-klp2r\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:09.128310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.128268    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" event={"ID":"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559","Type":"ContainerStarted","Data":"fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10"}
Nov 02 18:26:09.129988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.129967    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"
Nov 02 18:26:09.130119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.130002    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww" event={"ID":"33b075f3-7946-46f3-b9bb-b0b479bafa3c","Type":"ContainerDied","Data":"666a0c5a1bcfb7db47d2d65fb7b091444d5bf4e6bf24808684e69aa85dc2bdfb"}
Nov 02 18:26:09.130119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.130045    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="666a0c5a1bcfb7db47d2d65fb7b091444d5bf4e6bf24808684e69aa85dc2bdfb"
Nov 02 18:26:09.703865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.703346    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:26:09.704281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.704252    5042 patch_prober.go:29] interesting pod/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Nov 02 18:26:09.704439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:09.704312    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Nov 02 18:26:10.704167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:10.704126    5042 patch_prober.go:29] interesting pod/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Nov 02 18:26:10.704715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:10.704207    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Nov 02 18:26:10.914058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:10.909973    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:26:11.700748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:11.700705    5042 patch_prober.go:29] interesting pod/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Nov 02 18:26:11.701167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:11.701125    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Nov 02 18:26:12.703667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:12.703318    5042 patch_prober.go:29] interesting pod/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Nov 02 18:26:12.703667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:12.703372    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Nov 02 18:26:13.702342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:13.702301    5042 patch_prober.go:29] interesting pod/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Nov 02 18:26:13.702649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:13.702614    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Nov 02 18:26:14.698340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:14.698304    5042 patch_prober.go:29] interesting pod/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body=
Nov 02 18:26:14.698776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:14.698378    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused"
Nov 02 18:26:15.705821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:15.705788    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:26:15.707161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:15.707139    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:26:18.542059 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:26:18.541971    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:26:21.027612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:21.027576    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"]
Nov 02 18:26:21.036352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:21.036309    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426eb15-cbjkm"]
Nov 02 18:26:21.548305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:21.548269    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f96ead-b24e-490c-8420-e7a04d030652" path="/var/lib/kubelet/pods/01f96ead-b24e-490c-8420-e7a04d030652/volumes"
Nov 02 18:26:25.803826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:25.803785    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"]
Nov 02 18:26:25.804856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:25.804783    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy" containerID="cri-o://e2933dbb106f482dbc860c61ede7068bb975999a22d62a09ac36940d7b77ef08" gracePeriod=30
Nov 02 18:26:31.459131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.459097    5042 generic.go:356] "Generic (PLEG): container finished" podID="279aa536-5551-46b2-81e3-656424fe04ad" containerID="e2933dbb106f482dbc860c61ede7068bb975999a22d62a09ac36940d7b77ef08" exitCode=0
Nov 02 18:26:31.459615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.459171    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" event={"ID":"279aa536-5551-46b2-81e3-656424fe04ad","Type":"ContainerDied","Data":"e2933dbb106f482dbc860c61ede7068bb975999a22d62a09ac36940d7b77ef08"}
Nov 02 18:26:31.459615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.459254    5042 scope.go:122] "RemoveContainer" containerID="6a0e6016bf110e6f3457927e073b6cc59f80b2adc0c0ae8574057c800f6c34ac"
Nov 02 18:26:31.549080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.549051    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 18:26:31.632887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.632762    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.632969    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633017    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633053    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633096 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633090    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633131    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633162    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633198    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633280    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5\") pod \"279aa536-5551-46b2-81e3-656424fe04ad\" (UID: \"279aa536-5551-46b2-81e3-656424fe04ad\") "
Nov 02 18:26:31.633892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633796    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:26:31.634067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.633919    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:26:31.634483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.634443    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:26:31.635027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.634998    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:26:31.635547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.635517    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:26:31.636868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.636820    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:26:31.637535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.637508    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:31.637816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.637795    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "kube-api-access-q8bg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:26:31.638161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.638137    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo" pod "279aa536-5551-46b2-81e3-656424fe04ad" (UID: "279aa536-5551-46b2-81e3-656424fe04ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:26:31.735805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735764    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8bg5\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-kube-api-access-q8bg5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.735805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735799    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.735805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735808    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.736098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735817    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279aa536-5551-46b2-81e3-656424fe04ad-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.736098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735827    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.736098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735848    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.736098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735858    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279aa536-5551-46b2-81e3-656424fe04ad-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.736098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735866    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279aa536-5551-46b2-81e3-656424fe04ad-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:31.736098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:31.735874    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279aa536-5551-46b2-81e3-656424fe04ad-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:26:32.026166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.026126    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"]
Nov 02 18:26:32.030541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.030512    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6f59ca-vdccm"]
Nov 02 18:26:32.469412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.469374    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78" event={"ID":"279aa536-5551-46b2-81e3-656424fe04ad","Type":"ContainerDied","Data":"8b300906601cb0017b51d2d6fc25c4ecdbc750715e3e24681efb9c399e65d77b"}
Nov 02 18:26:32.469412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.469397    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"
Nov 02 18:26:32.469929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.469470    5042 scope.go:122] "RemoveContainer" containerID="e2933dbb106f482dbc860c61ede7068bb975999a22d62a09ac36940d7b77ef08"
Nov 02 18:26:32.500138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.500102    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"]
Nov 02 18:26:32.504436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.504403    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-24cfb6-openshift-default-59b8747df-29j78"]
Nov 02 18:26:32.588674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.587458    5042 scope.go:122] "RemoveContainer" containerID="1f5120357436a1b4d0c6ccc9196765b4e1d017c33f12d757391d52c373f1441e"
Nov 02 18:26:32.621169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:32.621143    5042 scope.go:122] "RemoveContainer" containerID="79a41f3f27f4fbb0e33bcbd9e20258e10f39d8b56b63ea766cca7338db350810"
Nov 02 18:26:33.545568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:33.545531    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e49cc75-fea8-4d97-b290-7922411562b3" path="/var/lib/kubelet/pods/1e49cc75-fea8-4d97-b290-7922411562b3/volumes"
Nov 02 18:26:33.546312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:33.546289    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279aa536-5551-46b2-81e3-656424fe04ad" path="/var/lib/kubelet/pods/279aa536-5551-46b2-81e3-656424fe04ad/volumes"
Nov 02 18:26:38.813937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:38.812170    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:38.919878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:38.919788    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.030867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.030020    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.093930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.092944    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.133867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.133005    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.494575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.492925    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.586339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.586296    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk" podUID="2803f60a-6320-44ec-8961-4a6f026d6a45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:26:39.823922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.812943    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.823922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.813601    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:26:39.950866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.948966    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:39.950866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:39.949644    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:26:40.019866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.019588    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:26:40.056240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.054108    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:26:40.056240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.054715    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:26:40.441241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.441193    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 02 18:26:40.560278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.559785    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded"
Nov 02 18:26:40.601904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.587066    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:40.630563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.630525    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:26:40.754924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.733123    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:26:40.977252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.977201    5042 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 02 18:26:40.990465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:40.980020    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Nov 02 18:26:41.120452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.120337    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Nov 02 18:26:41.218868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.145438    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:41.594418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.448062    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:41.594772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.594050    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:41.595515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.594188    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:26:41.691614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.691559    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:26:41.727514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.727484    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:26:41.735437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.735402    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:26:41.765882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.765828    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:26:41.773508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.773476    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:26:41.776741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:26:41.776716    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:26:48.545907 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:26:48.545870    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:27:05.733138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:05.733101    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"]
Nov 02 18:27:05.733646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:05.733317    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" podUID="ab49f445-8557-460c-9080-8caa6e601004" containerName="discovery" containerID="cri-o://13364444460ba3c8ce5f567351ef814670a11866142b71b413ceddf27b90fafa" gracePeriod=30
Nov 02 18:27:06.030343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.029372    5042 generic.go:356] "Generic (PLEG): container finished" podID="ab49f445-8557-460c-9080-8caa6e601004" containerID="13364444460ba3c8ce5f567351ef814670a11866142b71b413ceddf27b90fafa" exitCode=0
Nov 02 18:27:06.030343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.029760    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" event={"ID":"ab49f445-8557-460c-9080-8caa6e601004","Type":"ContainerDied","Data":"13364444460ba3c8ce5f567351ef814670a11866142b71b413ceddf27b90fafa"}
Nov 02 18:27:06.081787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.081751    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:27:06.152331 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152293    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.152525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152370    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.152525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152475    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.152652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152584    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.152749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152733    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.152800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152778    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.152920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.152900    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts\") pod \"ab49f445-8557-460c-9080-8caa6e601004\" (UID: \"ab49f445-8557-460c-9080-8caa6e601004\") "
Nov 02 18:27:06.154403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.154366    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:06.155917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.155875    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:06.156238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.156206    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:06.156432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.156408    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:06.158315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.158287    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:06.158417 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.158379    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:06.158417 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.158394    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964" pod "ab49f445-8557-460c-9080-8caa6e601004" (UID: "ab49f445-8557-460c-9080-8caa6e601004"). InnerVolumeSpecName "kube-api-access-fn964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:06.255320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255279    5042 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-dns-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:06.255320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255313    5042 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab49f445-8557-460c-9080-8caa6e601004-istio-csr-ca-configmap\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:06.255320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255323    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn964\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-kube-api-access-fn964\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:06.255591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255333    5042 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab49f445-8557-460c-9080-8caa6e601004-local-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:06.255591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255347    5042 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-cacerts\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:06.255591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255361    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab49f445-8557-460c-9080-8caa6e601004-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:06.255591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:06.255370    5042 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab49f445-8557-460c-9080-8caa6e601004-istio-kubeconfig\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:07.059865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:07.055143    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl" event={"ID":"ab49f445-8557-460c-9080-8caa6e601004","Type":"ContainerDied","Data":"0fad1d9c92b4a43a148bf0493024c79799ff43157080e6e4f7a273d52941cd86"}
Nov 02 18:27:07.059865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:07.055257    5042 scope.go:122] "RemoveContainer" containerID="13364444460ba3c8ce5f567351ef814670a11866142b71b413ceddf27b90fafa"
Nov 02 18:27:07.059865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:07.055431    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"
Nov 02 18:27:07.105150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:07.105113    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"]
Nov 02 18:27:07.110634 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:07.110598    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tw9kl"]
Nov 02 18:27:07.546927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:07.546896    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab49f445-8557-460c-9080-8caa6e601004" path="/var/lib/kubelet/pods/ab49f445-8557-460c-9080-8caa6e601004/volumes"
Nov 02 18:27:08.230033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.229997    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"]
Nov 02 18:27:08.230756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230732    5042 cpu_manager.go:414] "containerMap: removing container" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy"
Nov 02 18:27:08.230756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230752    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230764    5042 cpu_manager.go:414] "containerMap: removing container" podUID="33b075f3-7946-46f3-b9bb-b0b479bafa3c" containerName="revoke-keys"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230771    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b075f3-7946-46f3-b9bb-b0b479bafa3c" containerName="revoke-keys"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230807    5042 cpu_manager.go:414] "containerMap: removing container" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230812    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230822    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ab49f445-8557-460c-9080-8caa6e601004" containerName="discovery"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.230826    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab49f445-8557-460c-9080-8caa6e601004" containerName="discovery"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.231009    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.231025    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab49f445-8557-460c-9080-8caa6e601004" containerName="discovery"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.231043    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="279aa536-5551-46b2-81e3-656424fe04ad" containerName="istio-proxy"
Nov 02 18:27:08.231049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.231053    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b075f3-7946-46f3-b9bb-b0b479bafa3c" containerName="revoke-keys"
Nov 02 18:27:08.235439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.235405    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.244951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.244887    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"]
Nov 02 18:27:08.246352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.245181    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"kube-root-ca.crt\""
Nov 02 18:27:08.248163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.248137    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"openshift-service-ca.crt\""
Nov 02 18:27:08.248321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.248143    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"default-dockercfg-84qnw\""
Nov 02 18:27:08.248321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.248205    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-47e1c2\"/\"rate-test-model-47e1c2-kserve-self-signed-certs\""
Nov 02 18:27:08.292745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.292701    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.292965 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.292919    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.293062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.292971    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pnf\" (UniqueName: \"kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.293062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.293020    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.293195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.293097    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.293195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.293141    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.413957    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.414240    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.414322    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pnf\" (UniqueName: \"kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.414399    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.414489    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.414535    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.415246    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.415501    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.418870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.416161    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.458389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.451865    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pnf\" (UniqueName: \"kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.458389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.457214    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.476865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.473221    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm\") pod \"rate-test-model-47e1c2-kserve-cb766f557-mnwf4\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") " pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.562860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.562162    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:08.949376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:08.949344    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"]
Nov 02 18:27:08.950426 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:27:08.950400    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdeafd72_b1e2_478d_9aee_2ff97f377909.slice/crio-697319c860af1bcfecca885bf4c8ddffe35090af6ea26be373f3945cda594c04 WatchSource:0}: Error finding container 697319c860af1bcfecca885bf4c8ddffe35090af6ea26be373f3945cda594c04: Status 404 returned error can't find the container with id 697319c860af1bcfecca885bf4c8ddffe35090af6ea26be373f3945cda594c04
Nov 02 18:27:09.075482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:09.075438    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" event={"ID":"fdeafd72-b1e2-478d-9aee-2ff97f377909","Type":"ContainerStarted","Data":"2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67"}
Nov 02 18:27:09.075690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:09.075498    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" event={"ID":"fdeafd72-b1e2-478d-9aee-2ff97f377909","Type":"ContainerStarted","Data":"697319c860af1bcfecca885bf4c8ddffe35090af6ea26be373f3945cda594c04"}
Nov 02 18:27:09.098800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:09.098733    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" podStartSLOduration=1.098712206 podStartE2EDuration="1.098712206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:09.092734264 +0000 UTC m=+3241.672143649" watchObservedRunningTime="2026-11-02 18:27:09.098712206 +0000 UTC m=+3241.678121593"
Nov 02 18:27:09.564712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:09.564677    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:14.029786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:14.029746    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"]
Nov 02 18:27:14.034391 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:14.034365    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdca042-qvvlq"]
Nov 02 18:27:15.547317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:15.547285    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d666c202-2d1a-4e38-9d50-414071b52ef1" path="/var/lib/kubelet/pods/d666c202-2d1a-4e38-9d50-414071b52ef1/volumes"
Nov 02 18:27:18.571880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:18.570311    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:18.611371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:18.611340    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:18.613476 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:27:18.613445    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:27:19.184047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:19.184013    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:27:22.862319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.862286    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"]
Nov 02 18:27:22.868786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.868759    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:22.872627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.872599    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:27:22.873363 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.873335    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"]
Nov 02 18:27:22.873565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.873423    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:27:22.966340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.966303    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"]
Nov 02 18:27:22.967247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.967212    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:22.967470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.967415    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ngf\" (UniqueName: \"kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:22.967596 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.967525    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:22.981926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.981897    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:22.986261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.986038    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"kube-root-ca.crt\""
Nov 02 18:27:22.986261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.986221    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"rate-test-model-9165c3-kserve-self-signed-certs\""
Nov 02 18:27:22.988250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.988195    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"openshift-service-ca.crt\""
Nov 02 18:27:22.991210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:22.990679    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"]
Nov 02 18:27:23.068671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.068632    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.068898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.068729    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.068898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.068757    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.068898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.068795    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.068898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.068862    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.069142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.068948    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pln2\" (UniqueName: \"kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.069315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.069157    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ngf\" (UniqueName: \"kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.069315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.069198    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.069315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.069253    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.069763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.069645    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.072960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.072933    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.105867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.104108    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-9165c3\"/\"default-dockercfg-njhzl\""
Nov 02 18:27:23.139754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.139627    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ngf\" (UniqueName: \"kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf\") pod \"maas-api-revoke-sub-491e8b2b-4k8zv\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.188864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.171119    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pln2\" (UniqueName: \"kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.188864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.171286    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.188864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.171347    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.188864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.171485    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.188864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.171517    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.188864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.171570    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.193860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.193198    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.193860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.193487    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.198027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.197649    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pln2\" (UniqueName: \"kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.198184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.198119    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:23.198406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.172829    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.226100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.225162    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.314830 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.313088    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs\") pod \"rate-test-model-9165c3-kserve-7b4c94c7c-q64c4\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") " pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.608446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.608404    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:23.613532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.613506    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"]
Nov 02 18:27:23.911369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:23.911337    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"]
Nov 02 18:27:23.912395 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:27:23.912367    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded38f5f4_07bf_4fe1_bd65_3cb56a8106b3.slice/crio-2c294732ce51f51c54b8ab3f41fe8fcfdba6dd29f4f853862ebf70817c512e60 WatchSource:0}: Error finding container 2c294732ce51f51c54b8ab3f41fe8fcfdba6dd29f4f853862ebf70817c512e60: Status 404 returned error can't find the container with id 2c294732ce51f51c54b8ab3f41fe8fcfdba6dd29f4f853862ebf70817c512e60
Nov 02 18:27:24.294651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.294611    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv" event={"ID":"7caf1ff8-8099-40b9-aca9-0c2b70330b89","Type":"ContainerStarted","Data":"c84c0579baaa84d607b506475a9749175a08c7c69aec2825d2d5a7c1315ced2d"}
Nov 02 18:27:24.294927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.294682    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv" event={"ID":"7caf1ff8-8099-40b9-aca9-0c2b70330b89","Type":"ContainerStarted","Data":"06ac2633bc2ad6600be32e2237d07773f47839331d90bc544a47fb9ece59f446"}
Nov 02 18:27:24.296586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.296558    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4" event={"ID":"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3","Type":"ContainerStarted","Data":"1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f"}
Nov 02 18:27:24.296716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.296595    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4" event={"ID":"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3","Type":"ContainerStarted","Data":"2c294732ce51f51c54b8ab3f41fe8fcfdba6dd29f4f853862ebf70817c512e60"}
Nov 02 18:27:24.316242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.316192    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv" podStartSLOduration=2.316177186 podStartE2EDuration="2.316177186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:24.313977334 +0000 UTC m=+3256.893386719" watchObservedRunningTime="2026-11-02 18:27:24.316177186 +0000 UTC m=+3256.895586572"
Nov 02 18:27:24.331877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.331788    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4" podStartSLOduration=2.331771311 podStartE2EDuration="2.331771311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:24.330110014 +0000 UTC m=+3256.909519402" watchObservedRunningTime="2026-11-02 18:27:24.331771311 +0000 UTC m=+3256.911180689"
Nov 02 18:27:24.608657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:24.608623    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:25.308023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:25.307985    5042 generic.go:356] "Generic (PLEG): container finished" podID="7caf1ff8-8099-40b9-aca9-0c2b70330b89" containerID="c84c0579baaa84d607b506475a9749175a08c7c69aec2825d2d5a7c1315ced2d" exitCode=0
Nov 02 18:27:25.308528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:25.308071    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv" event={"ID":"7caf1ff8-8099-40b9-aca9-0c2b70330b89","Type":"ContainerDied","Data":"c84c0579baaa84d607b506475a9749175a08c7c69aec2825d2d5a7c1315ced2d"}
Nov 02 18:27:26.026432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.026398    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"]
Nov 02 18:27:26.040515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.040469    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41104cb-29r2k"]
Nov 02 18:27:26.520716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.520107    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:26.671997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.671911    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca\") pod \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") "
Nov 02 18:27:26.672161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.672113    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca\") pod \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") "
Nov 02 18:27:26.672221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.672159    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf\" (UniqueName: \"kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf\") pod \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\" (UID: \"7caf1ff8-8099-40b9-aca9-0c2b70330b89\") "
Nov 02 18:27:26.672550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.672515    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca" pod "7caf1ff8-8099-40b9-aca9-0c2b70330b89" (UID: "7caf1ff8-8099-40b9-aca9-0c2b70330b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:26.673095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.673073    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:26.674877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.674823    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca" pod "7caf1ff8-8099-40b9-aca9-0c2b70330b89" (UID: "7caf1ff8-8099-40b9-aca9-0c2b70330b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:26.675114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.675094    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf" pod "7caf1ff8-8099-40b9-aca9-0c2b70330b89" (UID: "7caf1ff8-8099-40b9-aca9-0c2b70330b89"). InnerVolumeSpecName "kube-api-access-m5ngf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:26.775337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.775298    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7caf1ff8-8099-40b9-aca9-0c2b70330b89-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:26.775337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:26.775338    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ngf\" (UniqueName: \"kubernetes.io/projected/7caf1ff8-8099-40b9-aca9-0c2b70330b89-kube-api-access-m5ngf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:27.331651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:27.331613    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv" event={"ID":"7caf1ff8-8099-40b9-aca9-0c2b70330b89","Type":"ContainerDied","Data":"06ac2633bc2ad6600be32e2237d07773f47839331d90bc544a47fb9ece59f446"}
Nov 02 18:27:27.331651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:27.331643    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"
Nov 02 18:27:27.332038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:27.331671    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ac2633bc2ad6600be32e2237d07773f47839331d90bc544a47fb9ece59f446"
Nov 02 18:27:27.545424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:27.545383    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414a1b16-0d3c-4ae8-8b83-26cafc1997b1" path="/var/lib/kubelet/pods/414a1b16-0d3c-4ae8-8b83-26cafc1997b1/volumes"
Nov 02 18:27:32.681945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:32.681906    5042 scope.go:122] "RemoveContainer" containerID="d41f81d11973ec26e0fb8910db50f035a0a939a53a7e4f10c1b341eadaadc681"
Nov 02 18:27:32.696014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:32.695988    5042 scope.go:122] "RemoveContainer" containerID="defe0c92aa6569adb30c0ab6346789fe124ac7903b0d01e1d557fc56ac747281"
Nov 02 18:27:33.608827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:33.608795    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:33.622013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:33.621979    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:34.036959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.036927    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"]
Nov 02 18:27:34.037949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.037926    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7caf1ff8-8099-40b9-aca9-0c2b70330b89" containerName="revoke-keys"
Nov 02 18:27:34.037949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.037949    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7caf1ff8-8099-40b9-aca9-0c2b70330b89" containerName="revoke-keys"
Nov 02 18:27:34.038679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.038132    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7caf1ff8-8099-40b9-aca9-0c2b70330b89" containerName="revoke-keys"
Nov 02 18:27:34.043685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.043661    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.047214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.047188    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:27:34.047357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.047198    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:27:34.049551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.049524    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"]
Nov 02 18:27:34.114649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.114562    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx854\" (UniqueName: \"kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.114812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.114767    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.115049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.115027    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.216814 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.216776    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.217052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.216965    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx854\" (UniqueName: \"kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.217115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.217099    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.217934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.217892    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.219610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.219585    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.225112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.225088    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx854\" (UniqueName: \"kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854\") pod \"maas-api-revoke-sub-b054f5a7-t7f68\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.363043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.362962    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:34.443871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.441283    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:27:34.736213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:34.736181    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"]
Nov 02 18:27:35.415663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:35.415476    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68" event={"ID":"1566fab0-31d1-4ca4-bed5-60a5da3b5a71","Type":"ContainerStarted","Data":"5fe3d44e6a200e8916c6cdd056aedc00cea35059c031550eafd67143d63290a0"}
Nov 02 18:27:35.415663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:35.415539    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68" event={"ID":"1566fab0-31d1-4ca4-bed5-60a5da3b5a71","Type":"ContainerStarted","Data":"ab6ac67d429e4560db57110c1a4c0c158e895ba74db53d2efed9cfe4114ee469"}
Nov 02 18:27:35.431151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:35.431098    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68" podStartSLOduration=1.4310838590000001 podStartE2EDuration="1.431083859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:35.430102427 +0000 UTC m=+3268.009511814" watchObservedRunningTime="2026-11-02 18:27:35.431083859 +0000 UTC m=+3268.010493245"
Nov 02 18:27:36.426324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:36.426288    5042 generic.go:356] "Generic (PLEG): container finished" podID="1566fab0-31d1-4ca4-bed5-60a5da3b5a71" containerID="5fe3d44e6a200e8916c6cdd056aedc00cea35059c031550eafd67143d63290a0" exitCode=0
Nov 02 18:27:36.426754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:36.426322    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68" event={"ID":"1566fab0-31d1-4ca4-bed5-60a5da3b5a71","Type":"ContainerDied","Data":"5fe3d44e6a200e8916c6cdd056aedc00cea35059c031550eafd67143d63290a0"}
Nov 02 18:27:37.699428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.698375    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:37.850741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.850706    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca\") pod \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") "
Nov 02 18:27:37.850975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.850960    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854\" (UniqueName: \"kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854\") pod \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") "
Nov 02 18:27:37.851018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.851005    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca\") pod \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\" (UID: \"1566fab0-31d1-4ca4-bed5-60a5da3b5a71\") "
Nov 02 18:27:37.851086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.851052    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca" pod "1566fab0-31d1-4ca4-bed5-60a5da3b5a71" (UID: "1566fab0-31d1-4ca4-bed5-60a5da3b5a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:37.852083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.851573    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:37.853441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.853411    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca" pod "1566fab0-31d1-4ca4-bed5-60a5da3b5a71" (UID: "1566fab0-31d1-4ca4-bed5-60a5da3b5a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:37.853668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.853649    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854" pod "1566fab0-31d1-4ca4-bed5-60a5da3b5a71" (UID: "1566fab0-31d1-4ca4-bed5-60a5da3b5a71"). InnerVolumeSpecName "kube-api-access-zx854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:37.953567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.953456    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx854\" (UniqueName: \"kubernetes.io/projected/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-kube-api-access-zx854\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:37.953567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:37.953497    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1566fab0-31d1-4ca4-bed5-60a5da3b5a71-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:38.444765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:38.444732    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"
Nov 02 18:27:38.445032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:38.444771    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68" event={"ID":"1566fab0-31d1-4ca4-bed5-60a5da3b5a71","Type":"ContainerDied","Data":"ab6ac67d429e4560db57110c1a4c0c158e895ba74db53d2efed9cfe4114ee469"}
Nov 02 18:27:38.445032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:38.444819    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab6ac67d429e4560db57110c1a4c0c158e895ba74db53d2efed9cfe4114ee469"
Nov 02 18:27:41.571388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:41.571353    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"]
Nov 02 18:27:41.589651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:41.589599    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"]
Nov 02 18:27:44.318166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:44.318117    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"]
Nov 02 18:27:44.333576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:44.318406    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg" podUID="afc6f1d9-87dc-454d-8951-55a5626905a0" containerName="maas-api" containerID="cri-o://f38cdc01ad48a1849a689ace11b03880b7fa2381426a961c8507e22f3a5b0e75" gracePeriod=30
Nov 02 18:27:44.937908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:44.929665    5042 generic.go:356] "Generic (PLEG): container finished" podID="afc6f1d9-87dc-454d-8951-55a5626905a0" containerID="f38cdc01ad48a1849a689ace11b03880b7fa2381426a961c8507e22f3a5b0e75" exitCode=0
Nov 02 18:27:44.937908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:44.929995    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg" event={"ID":"afc6f1d9-87dc-454d-8951-55a5626905a0","Type":"ContainerDied","Data":"f38cdc01ad48a1849a689ace11b03880b7fa2381426a961c8507e22f3a5b0e75"}
Nov 02 18:27:45.210864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.210772    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:27:45.282016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.281984    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca\") pod \"afc6f1d9-87dc-454d-8951-55a5626905a0\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") "
Nov 02 18:27:45.282216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.282045    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs\") pod \"afc6f1d9-87dc-454d-8951-55a5626905a0\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") "
Nov 02 18:27:45.282216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.282092    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls\") pod \"afc6f1d9-87dc-454d-8951-55a5626905a0\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") "
Nov 02 18:27:45.282326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.282230    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6\" (UniqueName: \"kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6\") pod \"afc6f1d9-87dc-454d-8951-55a5626905a0\" (UID: \"afc6f1d9-87dc-454d-8951-55a5626905a0\") "
Nov 02 18:27:45.282512 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.282483    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca" pod "afc6f1d9-87dc-454d-8951-55a5626905a0" (UID: "afc6f1d9-87dc-454d-8951-55a5626905a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:45.283712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.283182    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc6f1d9-87dc-454d-8951-55a5626905a0-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:45.285887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.284750    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs" pod "afc6f1d9-87dc-454d-8951-55a5626905a0" (UID: "afc6f1d9-87dc-454d-8951-55a5626905a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:45.285887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.284940    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls" pod "afc6f1d9-87dc-454d-8951-55a5626905a0" (UID: "afc6f1d9-87dc-454d-8951-55a5626905a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:45.285887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.285256    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6" pod "afc6f1d9-87dc-454d-8951-55a5626905a0" (UID: "afc6f1d9-87dc-454d-8951-55a5626905a0"). InnerVolumeSpecName "kube-api-access-jf6j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:45.385435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.385398    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:45.385435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.385432    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afc6f1d9-87dc-454d-8951-55a5626905a0-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:45.385435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.385443    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf6j6\" (UniqueName: \"kubernetes.io/projected/afc6f1d9-87dc-454d-8951-55a5626905a0-kube-api-access-jf6j6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:45.941489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.941454    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"
Nov 02 18:27:45.941489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.941479    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg" event={"ID":"afc6f1d9-87dc-454d-8951-55a5626905a0","Type":"ContainerDied","Data":"4627cc9e9b1e9957cfcb585e0a9cda66db7e98b59ccf536e6080cc68a4034c4d"}
Nov 02 18:27:45.941721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.941554    5042 scope.go:122] "RemoveContainer" containerID="f38cdc01ad48a1849a689ace11b03880b7fa2381426a961c8507e22f3a5b0e75"
Nov 02 18:27:45.965137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.965091    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"]
Nov 02 18:27:45.970792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:45.970763    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8a098971-75499b458c-c2vtg"]
Nov 02 18:27:46.574527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.574479    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl" podUID="c92574ad-87f5-408e-a4dc-1b1c46bc69a6" containerName="payload-processing" containerID="cri-o://b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d" gracePeriod=25
Nov 02 18:27:46.591179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.591113    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd" podUID="ed73717a-e17c-4cb4-8b93-57e43a0bc919" containerName="payload-pre-processing" containerID="cri-o://c58175d164acc78afdc570038688729b1fdc4ab3ae9ad1de7fdf8a5a233030cc" gracePeriod=25
Nov 02 18:27:46.917594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.917551    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:27:46.957456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.957418    5042 generic.go:356] "Generic (PLEG): container finished" podID="c92574ad-87f5-408e-a4dc-1b1c46bc69a6" containerID="b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d" exitCode=0
Nov 02 18:27:46.957657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.957475    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"
Nov 02 18:27:46.957657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.957548    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl" event={"ID":"c92574ad-87f5-408e-a4dc-1b1c46bc69a6","Type":"ContainerDied","Data":"b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d"}
Nov 02 18:27:46.957657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.957600    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl" event={"ID":"c92574ad-87f5-408e-a4dc-1b1c46bc69a6","Type":"ContainerDied","Data":"5694a25119fdcb93ea862e87eb1d58e8f5db69ccc4b124ccd420a24613653fa8"}
Nov 02 18:27:46.957657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.957632    5042 scope.go:122] "RemoveContainer" containerID="b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d"
Nov 02 18:27:46.959528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.959506    5042 generic.go:356] "Generic (PLEG): container finished" podID="ed73717a-e17c-4cb4-8b93-57e43a0bc919" containerID="c58175d164acc78afdc570038688729b1fdc4ab3ae9ad1de7fdf8a5a233030cc" exitCode=0
Nov 02 18:27:46.959652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.959550    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd" event={"ID":"ed73717a-e17c-4cb4-8b93-57e43a0bc919","Type":"ContainerDied","Data":"c58175d164acc78afdc570038688729b1fdc4ab3ae9ad1de7fdf8a5a233030cc"}
Nov 02 18:27:46.959652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.959619    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd" event={"ID":"ed73717a-e17c-4cb4-8b93-57e43a0bc919","Type":"ContainerDied","Data":"affc70d9d3a9adcf523a014d1e3aa9f5de4f93ce724ee775980e2e0acb9aebf6"}
Nov 02 18:27:46.959652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.959648    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="affc70d9d3a9adcf523a014d1e3aa9f5de4f93ce724ee775980e2e0acb9aebf6"
Nov 02 18:27:46.966164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.966140    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:27:46.973974 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.973911    5042 scope.go:122] "RemoveContainer" containerID="b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d"
Nov 02 18:27:46.974261 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:27:46.974239    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d\": container with ID starting with b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d not found: ID does not exist" containerID="b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d"
Nov 02 18:27:46.974333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:46.974271    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d"} err="failed to get container status \"b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d\": rpc error: code = NotFound desc = could not find container \"b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d\": container with ID starting with b4e2b97b7dba8e8f6cb562b068cb6ba026924958f8dc7454aaa2e73907c1144d not found: ID does not exist"
Nov 02 18:27:47.029787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.029746    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume\") pod \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") "
Nov 02 18:27:47.030061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.030039    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7\" (UniqueName: \"kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7\") pod \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\" (UID: \"c92574ad-87f5-408e-a4dc-1b1c46bc69a6\") "
Nov 02 18:27:47.030220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.030202    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume\") pod \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") "
Nov 02 18:27:47.031393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.030239    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume" pod "c92574ad-87f5-408e-a4dc-1b1c46bc69a6" (UID: "c92574ad-87f5-408e-a4dc-1b1c46bc69a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:47.031393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.030275    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6\") pod \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\" (UID: \"ed73717a-e17c-4cb4-8b93-57e43a0bc919\") "
Nov 02 18:27:47.031393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.030719    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume" pod "ed73717a-e17c-4cb4-8b93-57e43a0bc919" (UID: "ed73717a-e17c-4cb4-8b93-57e43a0bc919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:47.031393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.031242    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.031393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.031269    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed73717a-e17c-4cb4-8b93-57e43a0bc919-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.032819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.032796    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7" pod "c92574ad-87f5-408e-a4dc-1b1c46bc69a6" (UID: "c92574ad-87f5-408e-a4dc-1b1c46bc69a6"). InnerVolumeSpecName "kube-api-access-2cdj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:47.033242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.033220    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6" pod "ed73717a-e17c-4cb4-8b93-57e43a0bc919" (UID: "ed73717a-e17c-4cb4-8b93-57e43a0bc919"). InnerVolumeSpecName "kube-api-access-bfpn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:47.132463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.132363    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpn6\" (UniqueName: \"kubernetes.io/projected/ed73717a-e17c-4cb4-8b93-57e43a0bc919-kube-api-access-bfpn6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.132463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.132402    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cdj7\" (UniqueName: \"kubernetes.io/projected/c92574ad-87f5-408e-a4dc-1b1c46bc69a6-kube-api-access-2cdj7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:47.286971 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.286932    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"]
Nov 02 18:27:47.291218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.291186    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8a098971-569568574f-d27dl"]
Nov 02 18:27:47.545704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.545672    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afc6f1d9-87dc-454d-8951-55a5626905a0" path="/var/lib/kubelet/pods/afc6f1d9-87dc-454d-8951-55a5626905a0/volumes"
Nov 02 18:27:47.546132 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.546116    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92574ad-87f5-408e-a4dc-1b1c46bc69a6" path="/var/lib/kubelet/pods/c92574ad-87f5-408e-a4dc-1b1c46bc69a6/volumes"
Nov 02 18:27:47.970793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.970763    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"
Nov 02 18:27:47.996491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:47.996457    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"]
Nov 02 18:27:48.002493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:48.002458    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8a098971-57dd9c6d75-566nd"]
Nov 02 18:27:48.579156 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:27:48.579113    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:27:49.338807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:49.338764    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"]
Nov 02 18:27:49.339316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:49.338996    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy" containerID="cri-o://fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10" gracePeriod=30
Nov 02 18:27:49.640292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:49.640203    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed73717a-e17c-4cb4-8b93-57e43a0bc919" path="/var/lib/kubelet/pods/ed73717a-e17c-4cb4-8b93-57e43a0bc919/volumes"
Nov 02 18:27:54.683420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.683382    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:27:54.738355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738308    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.738570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738382    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.738570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738508    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.738570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738548    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.738794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738772    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.738910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738833    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.738910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.738896    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.739042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.739003    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.739102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.739051    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data\") pod \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\" (UID: \"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559\") "
Nov 02 18:27:54.740128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.740090    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.740317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.740295    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.741496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.740816    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.741496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.741112    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.741496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.741411    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:54.743006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.742971    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "kube-api-access-6m7tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:54.743137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.743073    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:54.744546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.744520    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:54.744714 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.744697    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy" pod "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" (UID: "fb0f812c-97f1-4a4d-9ee3-9d135fd3b559"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:54.841695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841659    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841691    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841700    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841713    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m7tz\" (UniqueName: \"kubernetes.io/projected/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-kube-api-access-6m7tz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841722    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841732    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841741    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841749    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:54.841995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:54.841758    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:27:55.085068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.085024    5042 generic.go:356] "Generic (PLEG): container finished" podID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerID="fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10" exitCode=0
Nov 02 18:27:55.085289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.085088    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"
Nov 02 18:27:55.085289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.085103    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" event={"ID":"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559","Type":"ContainerDied","Data":"fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10"}
Nov 02 18:27:55.085289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.085169    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68" event={"ID":"fb0f812c-97f1-4a4d-9ee3-9d135fd3b559","Type":"ContainerDied","Data":"07d9088d03ed7362254986e35b7f32513f5cb01ec6e8e240176fb9fbba9697e1"}
Nov 02 18:27:55.085289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.085199    5042 scope.go:122] "RemoveContainer" containerID="fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10"
Nov 02 18:27:55.168063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.168031    5042 scope.go:122] "RemoveContainer" containerID="1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"
Nov 02 18:27:55.176871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.174023    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"]
Nov 02 18:27:55.181074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.181044    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8a098971-openshift-default-6bb9f7f49-nzq68"]
Nov 02 18:27:55.189346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.189312    5042 scope.go:122] "RemoveContainer" containerID="fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10"
Nov 02 18:27:55.189798 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:27:55.189767    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10\": container with ID starting with fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10 not found: ID does not exist" containerID="fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10"
Nov 02 18:27:55.189931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.189810    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10"} err="failed to get container status \"fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10\": rpc error: code = NotFound desc = could not find container \"fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10\": container with ID starting with fb6093265f22abe891b35ed48c936701675a5bd20c233ac7703faa6282455b10 not found: ID does not exist"
Nov 02 18:27:55.189931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.189917    5042 scope.go:122] "RemoveContainer" containerID="1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"
Nov 02 18:27:55.190332 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:27:55.190306    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a\": container with ID starting with 1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a not found: ID does not exist" containerID="1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"
Nov 02 18:27:55.190409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.190344    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a"} err="failed to get container status \"1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a\": rpc error: code = NotFound desc = could not find container \"1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a\": container with ID starting with 1b060564b034e1ae5fcd4a4091d2ced61c7ac69aa494ea026d12d2915c3a987a not found: ID does not exist"
Nov 02 18:27:55.546991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:55.546946    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" path="/var/lib/kubelet/pods/fb0f812c-97f1-4a4d-9ee3-9d135fd3b559/volumes"
Nov 02 18:27:56.527502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:56.527468    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"]
Nov 02 18:27:56.527993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:27:56.527689    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy" containerID="cri-o://d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a" gracePeriod=30
Nov 02 18:28:02.228906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.225910    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313316    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313431    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313500    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313618    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313687    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313716    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313815    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313938    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.317157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.313988    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo\") pod \"cca681ab-7126-4c85-9246-38815d12500f\" (UID: \"cca681ab-7126-4c85-9246-38815d12500f\") "
Nov 02 18:28:02.327050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.323008    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:02.327050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.324751    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:02.327050 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.317940    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:02.334976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.334898    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:02.334976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.334898    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "kube-api-access-7vlls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:02.335238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.334973    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:02.335238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.335086    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:02.337123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.337040    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:02.343305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.341657    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo" pod "cca681ab-7126-4c85-9246-38815d12500f" (UID: "cca681ab-7126-4c85-9246-38815d12500f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:28:02.343305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.342500    5042 generic.go:356] "Generic (PLEG): container finished" podID="cca681ab-7126-4c85-9246-38815d12500f" containerID="d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a" exitCode=0
Nov 02 18:28:02.343305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.342819    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" event={"ID":"cca681ab-7126-4c85-9246-38815d12500f","Type":"ContainerDied","Data":"d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a"}
Nov 02 18:28:02.343305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.342900    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5" event={"ID":"cca681ab-7126-4c85-9246-38815d12500f","Type":"ContainerDied","Data":"5fc572af64570fc929d6903ec81d229376784aa214f41bc66e1f1e7e0e3e9dbb"}
Nov 02 18:28:02.343305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.342940    5042 scope.go:122] "RemoveContainer" containerID="d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a"
Nov 02 18:28:02.343305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.343114    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"
Nov 02 18:28:02.443442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.443404    5042 scope.go:122] "RemoveContainer" containerID="d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a"
Nov 02 18:28:02.443991 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:02.443927    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a\": container with ID starting with d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a not found: ID does not exist" containerID="d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a"
Nov 02 18:28:02.444154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.443978    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a"} err="failed to get container status \"d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a\": rpc error: code = NotFound desc = could not find container \"d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a\": container with ID starting with d203b400461dafae1ec8d7c6d1fd683fcad9670f4083a00c13a02f1cdbd39b0a not found: ID does not exist"
Nov 02 18:28:02.446116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446085    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca681ab-7126-4c85-9246-38815d12500f-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446120    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca681ab-7126-4c85-9246-38815d12500f-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446137    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446162    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446176    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlls\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-kube-api-access-7vlls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446216    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca681ab-7126-4c85-9246-38815d12500f-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446235    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446251    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.446263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.446266    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca681ab-7126-4c85-9246-38815d12500f-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:02.465930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.454701    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"]
Nov 02 18:28:02.465930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:02.462710    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b6b6f4-openshift-default-786ffdb75c-8bvl5"]
Nov 02 18:28:03.550941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:03.550904    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca681ab-7126-4c85-9246-38815d12500f" path="/var/lib/kubelet/pods/cca681ab-7126-4c85-9246-38815d12500f/volumes"
Nov 02 18:28:06.506781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:06.506740    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"]
Nov 02 18:28:06.507646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:06.507609    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy" containerID="cri-o://403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132" gracePeriod=30
Nov 02 18:28:09.795413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:09.795383    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:28:09.798107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:09.798079    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:28:09.799148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:09.799130    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:28:09.801438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:09.801422    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:28:11.848308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.848275    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:28:11.961087 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961050    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961118    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961169    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961244    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961331    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961396    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:11.961500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961443    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961543    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961580    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961612    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:11.961919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961819    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket\") pod \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\" (UID: \"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83\") "
Nov 02 18:28:11.961997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.961967    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:11.962951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.962654    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:11.962951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.962681    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:11.962951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.962697    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:11.963456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.963275    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:11.963649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.963602    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:11.966171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.966132    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:11.966926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.966827    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:11.966926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.966877    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "kube-api-access-bd44c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:11.967122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:11.967028    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo" pod "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" (UID: "f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:28:12.064358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.064271    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:12.064358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.064302    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:12.064358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.064312    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:12.064358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.064323    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:12.064358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.064339    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd44c\" (UniqueName: \"kubernetes.io/projected/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-kube-api-access-bd44c\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:12.064358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.064352    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:12.479110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.479068    5042 generic.go:356] "Generic (PLEG): container finished" podID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerID="403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132" exitCode=0
Nov 02 18:28:12.479278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.479162    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" event={"ID":"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83","Type":"ContainerDied","Data":"403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132"}
Nov 02 18:28:12.479278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.479172    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"
Nov 02 18:28:12.479278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.479222    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl" event={"ID":"f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83","Type":"ContainerDied","Data":"b4c600adcac71572cfb4602adfd2898b44feb2a1447027f48e33002cb8527028"}
Nov 02 18:28:12.479278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.479252    5042 scope.go:122] "RemoveContainer" containerID="403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132"
Nov 02 18:28:12.495187 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.495163    5042 scope.go:122] "RemoveContainer" containerID="e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"
Nov 02 18:28:12.508201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.508172    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"]
Nov 02 18:28:12.511595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.511527    5042 scope.go:122] "RemoveContainer" containerID="403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132"
Nov 02 18:28:12.512049 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:12.512023    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132\": container with ID starting with 403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132 not found: ID does not exist" containerID="403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132"
Nov 02 18:28:12.512160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.512060    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132"} err="failed to get container status \"403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132\": rpc error: code = NotFound desc = could not find container \"403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132\": container with ID starting with 403fd5e402855002682407c94d4cc6fbc7332e84ff0ea812a5af92de6c0b7132 not found: ID does not exist"
Nov 02 18:28:12.512160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.512087    5042 scope.go:122] "RemoveContainer" containerID="e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"
Nov 02 18:28:12.512470 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:12.512442    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8\": container with ID starting with e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8 not found: ID does not exist" containerID="e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"
Nov 02 18:28:12.512538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.512478    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8"} err="failed to get container status \"e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8\": rpc error: code = NotFound desc = could not find container \"e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8\": container with ID starting with e1f46a2f4bf8c3cec36324fb0a5e2457209651ef4c9281bce4bc0b247cf5dff8 not found: ID does not exist"
Nov 02 18:28:12.513142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:12.513123    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2966e7-openshift-default-57b4d978b9-w5nzl"]
Nov 02 18:28:13.545244 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:13.545212    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" path="/var/lib/kubelet/pods/f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83/volumes"
Nov 02 18:28:18.652865 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:18.649570    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:28:26.504584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:26.504537    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"]
Nov 02 18:28:26.515815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:26.505500    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" podUID="530814f9-6689-4091-a006-a7f973b1395f" containerName="maas-api" containerID="cri-o://ce2acf928141597129502884770cf6b75064c9dff34d01a098506d0e807a16b2" gracePeriod=30
Nov 02 18:28:27.008862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.006337    5042 generic.go:356] "Generic (PLEG): container finished" podID="530814f9-6689-4091-a006-a7f973b1395f" containerID="ce2acf928141597129502884770cf6b75064c9dff34d01a098506d0e807a16b2" exitCode=0
Nov 02 18:28:27.008862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.006547    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" event={"ID":"530814f9-6689-4091-a006-a7f973b1395f","Type":"ContainerDied","Data":"ce2acf928141597129502884770cf6b75064c9dff34d01a098506d0e807a16b2"}
Nov 02 18:28:27.155108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.155075    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:28:27.229110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.229067    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2\" (UniqueName: \"kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2\") pod \"530814f9-6689-4091-a006-a7f973b1395f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") "
Nov 02 18:28:27.229320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.229215    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca\") pod \"530814f9-6689-4091-a006-a7f973b1395f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") "
Nov 02 18:28:27.229320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.229242    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs\") pod \"530814f9-6689-4091-a006-a7f973b1395f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") "
Nov 02 18:28:27.229320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.229301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\") pod \"530814f9-6689-4091-a006-a7f973b1395f\" (UID: \"530814f9-6689-4091-a006-a7f973b1395f\") "
Nov 02 18:28:27.229630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.229604    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca" pod "530814f9-6689-4091-a006-a7f973b1395f" (UID: "530814f9-6689-4091-a006-a7f973b1395f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:27.230160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.230097    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/530814f9-6689-4091-a006-a7f973b1395f-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:27.231760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.231734    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2" pod "530814f9-6689-4091-a006-a7f973b1395f" (UID: "530814f9-6689-4091-a006-a7f973b1395f"). InnerVolumeSpecName "kube-api-access-dfcj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:27.231909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.231788    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs" pod "530814f9-6689-4091-a006-a7f973b1395f" (UID: "530814f9-6689-4091-a006-a7f973b1395f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:27.231909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.231817    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls" pod "530814f9-6689-4091-a006-a7f973b1395f" (UID: "530814f9-6689-4091-a006-a7f973b1395f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:27.331721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.331683    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:27.331721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.331713    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfcj2\" (UniqueName: \"kubernetes.io/projected/530814f9-6689-4091-a006-a7f973b1395f-kube-api-access-dfcj2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:27.331721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:27.331726    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/530814f9-6689-4091-a006-a7f973b1395f-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:28.020372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:28.020320    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"
Nov 02 18:28:28.020904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:28.020327    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f" event={"ID":"530814f9-6689-4091-a006-a7f973b1395f","Type":"ContainerDied","Data":"2c6f68195a06b3e64c4cfdbdd7ac647b1cfe0bd29929a8c3a69a0516e48f802f"}
Nov 02 18:28:28.020904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:28.020479    5042 scope.go:122] "RemoveContainer" containerID="ce2acf928141597129502884770cf6b75064c9dff34d01a098506d0e807a16b2"
Nov 02 18:28:28.041373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:28.041343    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"]
Nov 02 18:28:28.045398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:28.045372    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc0b41-7c89cc84cd-njt5f"]
Nov 02 18:28:29.546491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:29.546440    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530814f9-6689-4091-a006-a7f973b1395f" path="/var/lib/kubelet/pods/530814f9-6689-4091-a006-a7f973b1395f/volumes"
Nov 02 18:28:32.739371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:32.739334    5042 scope.go:122] "RemoveContainer" containerID="ebdcbd979d9bdb137f1d44a4d64731a0eee547710173ef84a9a22ae8f0bcc0e7"
Nov 02 18:28:32.751239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:32.751213    5042 scope.go:122] "RemoveContainer" containerID="ab0f2a76c4ab13795bf5acf06e3fdd771880b89d1eee587c13080ceec6a12085"
Nov 02 18:28:36.473568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:36.473527    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"]
Nov 02 18:28:36.474021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:36.473833    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy" containerID="cri-o://b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905" gracePeriod=30
Nov 02 18:28:41.864966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.864934    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:28:41.941389 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941348    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.941612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941450    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.941612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941607    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.941758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941713    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.941819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941760    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.941908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941815    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.941908 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941873    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.942026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941929    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.942026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.941973    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token\") pod \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\" (UID: \"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1\") "
Nov 02 18:28:41.958042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.942505    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:41.958042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.945523    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:41.958042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.951518    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:41.958042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.953807    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:41.958042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.953894    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:41.959889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.959831    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "kube-api-access-4mhbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:41.960826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.960793    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:41.961820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.961782    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:41.974426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:41.974381    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo" pod "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" (UID: "f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:28:42.025656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.025613    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"]
Nov 02 18:28:42.030066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.030029    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-668abeec-sxtpd"]
Nov 02 18:28:42.055423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055378    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055419    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055435    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055449    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055463    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055476    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055489    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055502    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.055701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.055515    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhbv\" (UniqueName: \"kubernetes.io/projected/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1-kube-api-access-4mhbv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:42.196360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.196323    5042 generic.go:356] "Generic (PLEG): container finished" podID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerID="b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905" exitCode=0
Nov 02 18:28:42.196589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.196406    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"
Nov 02 18:28:42.196589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.196412    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" event={"ID":"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1","Type":"ContainerDied","Data":"b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905"}
Nov 02 18:28:42.196589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.196522    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm" event={"ID":"f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1","Type":"ContainerDied","Data":"50e2086ec00e9060523b22bb0a584679463263398228491918f6eb7cd965f3b8"}
Nov 02 18:28:42.196589 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.196554    5042 scope.go:122] "RemoveContainer" containerID="b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905"
Nov 02 18:28:42.221167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.221136    5042 scope.go:122] "RemoveContainer" containerID="a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"
Nov 02 18:28:42.230105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.230076    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"]
Nov 02 18:28:42.234234 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.234197    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cc0b41-openshift-default-6d5d745895-s74xm"]
Nov 02 18:28:42.236616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.236598    5042 scope.go:122] "RemoveContainer" containerID="b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905"
Nov 02 18:28:42.237029 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:42.237007    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905\": container with ID starting with b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905 not found: ID does not exist" containerID="b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905"
Nov 02 18:28:42.237124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.237041    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905"} err="failed to get container status \"b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905\": rpc error: code = NotFound desc = could not find container \"b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905\": container with ID starting with b8c71675618bb8fe273988960a87e6f5354b8289811e88c7d598e03b60a17905 not found: ID does not exist"
Nov 02 18:28:42.237124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.237061    5042 scope.go:122] "RemoveContainer" containerID="a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"
Nov 02 18:28:42.237360 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:42.237344    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8\": container with ID starting with a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8 not found: ID does not exist" containerID="a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"
Nov 02 18:28:42.237416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:42.237366    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8"} err="failed to get container status \"a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8\": rpc error: code = NotFound desc = could not find container \"a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8\": container with ID starting with a86f548164230dddb084d7d873b171714316e56aed86acf55b4b96681cfd64f8 not found: ID does not exist"
Nov 02 18:28:43.249459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:43.249409    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"]
Nov 02 18:28:43.263878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:43.263822    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"]
Nov 02 18:28:43.618970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:43.613712    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d6878e-1294-4f77-8aa5-f963d13c423c" path="/var/lib/kubelet/pods/12d6878e-1294-4f77-8aa5-f963d13c423c/volumes"
Nov 02 18:28:43.620939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:43.620903    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" path="/var/lib/kubelet/pods/f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1/volumes"
Nov 02 18:28:47.347977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:47.347940    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"]
Nov 02 18:28:47.348468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:47.348212    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy" containerID="cri-o://b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d" gracePeriod=30
Nov 02 18:28:48.250790 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:48.250744    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerName="payload-processing" containerID="cri-o://af0a9718ef882e647babae4b35065357c8e7411679390981fd7e45ad8ea2524a" gracePeriod=25
Nov 02 18:28:48.264561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:48.264501    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerName="payload-pre-processing" containerID="cri-o://efcbbddda0ca2a279f6d7c6f23e0d3d145334a146c12c8bec2f62bf423014e5b" gracePeriod=25
Nov 02 18:28:48.590339 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:48.590243    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:28:49.233863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:49.226298    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"]
Nov 02 18:28:49.233863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:49.226552    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy" containerID="cri-o://d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6" gracePeriod=30
Nov 02 18:28:52.860919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.857755    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:28:52.920521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.920488    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.920860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.920818    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.921033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.921017    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.921164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.921149    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.921324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.921310    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.921518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.921503    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.921647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.921633    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.921984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.921965    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.922263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.922245    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert\") pod \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\" (UID: \"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78\") "
Nov 02 18:28:52.923513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.923373    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:52.924117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.924046    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:52.924117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.924073    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:52.924914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.924892    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:52.925191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.925161    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:52.929569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.929527    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "kube-api-access-hfdfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:52.933961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.930036    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:52.934259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.934226    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:28:52.934396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:52.934312    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token" pod "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" (UID: "db1159bd-e7f6-4ba4-9c5e-b1e66e174f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024146    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024178    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024190    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfdfl\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-kube-api-access-hfdfl\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024199    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024208    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024216    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024224    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024233    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.024246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.024240    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:53.320811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.320719    5042 generic.go:356] "Generic (PLEG): container finished" podID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerID="b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d" exitCode=0
Nov 02 18:28:53.321008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.320822    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"
Nov 02 18:28:53.321008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.320881    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" event={"ID":"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78","Type":"ContainerDied","Data":"b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d"}
Nov 02 18:28:53.321008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.320938    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2" event={"ID":"db1159bd-e7f6-4ba4-9c5e-b1e66e174f78","Type":"ContainerDied","Data":"8d66b2ae19d85dcbf4a0dc4901792053a3daf76968354dcb13bebd6325751b7e"}
Nov 02 18:28:53.321008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.320968    5042 scope.go:122] "RemoveContainer" containerID="b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d"
Nov 02 18:28:53.333773 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.333748    5042 scope.go:122] "RemoveContainer" containerID="b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d"
Nov 02 18:28:53.334265 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:53.334217    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d\": container with ID starting with b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d not found: ID does not exist" containerID="b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d"
Nov 02 18:28:53.334436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.334262    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d"} err="failed to get container status \"b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d\": rpc error: code = NotFound desc = could not find container \"b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d\": container with ID starting with b75a733aeac6bad72c00452316eebfc35db97e7693aaa00526d7067cdd1e108d not found: ID does not exist"
Nov 02 18:28:53.351566 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.348122    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"]
Nov 02 18:28:53.352338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.352314    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bee8f6-openshift-default-58f76f6545-lndg2"]
Nov 02 18:28:53.546174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:53.546138    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" path="/var/lib/kubelet/pods/db1159bd-e7f6-4ba4-9c5e-b1e66e174f78/volumes"
Nov 02 18:28:54.572352 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.572313    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661337    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661419    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661538    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661610    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661655    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661696    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661746    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.661833    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.663338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.662057    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf\") pod \"a3249565-04a0-496f-92c1-afe90b3c5c90\" (UID: \"a3249565-04a0-496f-92c1-afe90b3c5c90\") "
Nov 02 18:28:54.665383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.665252    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:54.665733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.665555    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:54.667306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.667205    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:54.668268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.668232    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:54.669445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.668733    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:54.669703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.669659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:54.670267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.669891    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:54.673599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.671414    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "kube-api-access-h6fzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:54.673599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.673206    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo" pod "a3249565-04a0-496f-92c1-afe90b3c5c90" (UID: "a3249565-04a0-496f-92c1-afe90b3c5c90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:28:54.767273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767231    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6fzf\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-kube-api-access-h6fzf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767260    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767271    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767280    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767290    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767301    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767314    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767327    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3249565-04a0-496f-92c1-afe90b3c5c90-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:54.767616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:54.767335    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3249565-04a0-496f-92c1-afe90b3c5c90-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:28:55.028113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.028074    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:55.028338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.028074    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:28:55.379228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.379126    5042 generic.go:356] "Generic (PLEG): container finished" podID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerID="d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6" exitCode=0
Nov 02 18:28:55.379228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.379181    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"
Nov 02 18:28:55.379228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.379211    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" event={"ID":"a3249565-04a0-496f-92c1-afe90b3c5c90","Type":"ContainerDied","Data":"d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6"}
Nov 02 18:28:55.379531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.379270    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch" event={"ID":"a3249565-04a0-496f-92c1-afe90b3c5c90","Type":"ContainerDied","Data":"de5b6d2d34a871f360809288dd9b5921c1ded09f29665b453124d060d6d66e22"}
Nov 02 18:28:55.379531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.379292    5042 scope.go:122] "RemoveContainer" containerID="d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6"
Nov 02 18:28:55.394180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.394156    5042 scope.go:122] "RemoveContainer" containerID="d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6"
Nov 02 18:28:55.394537 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:28:55.394515    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6\": container with ID starting with d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6 not found: ID does not exist" containerID="d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6"
Nov 02 18:28:55.394614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.394545    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6"} err="failed to get container status \"d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6\": rpc error: code = NotFound desc = could not find container \"d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6\": container with ID starting with d2a0e64a757bc1830e6958a9a0476709e6c4ccb52f0a561985f7cb1b2f97e5c6 not found: ID does not exist"
Nov 02 18:28:55.409007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.408927    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"]
Nov 02 18:28:55.454865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.447853    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b1a788-openshift-default-f86c5f547-x9mch"]
Nov 02 18:28:55.545615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:28:55.545585    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" path="/var/lib/kubelet/pods/a3249565-04a0-496f-92c1-afe90b3c5c90/volumes"
Nov 02 18:29:05.035870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:05.034295    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:05.035870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:05.035138    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:08.672716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.672676    5042 generic.go:356] "Generic (PLEG): container finished" podID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerID="af0a9718ef882e647babae4b35065357c8e7411679390981fd7e45ad8ea2524a" exitCode=0
Nov 02 18:29:08.673190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.672945    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" event={"ID":"54e8c557-0b90-4592-b388-fc08b2fcc5b5","Type":"ContainerDied","Data":"af0a9718ef882e647babae4b35065357c8e7411679390981fd7e45ad8ea2524a"}
Nov 02 18:29:08.675785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.675651    5042 generic.go:356] "Generic (PLEG): container finished" podID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerID="efcbbddda0ca2a279f6d7c6f23e0d3d145334a146c12c8bec2f62bf423014e5b" exitCode=0
Nov 02 18:29:08.675785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.675723    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" event={"ID":"01716d13-d310-4be8-8c2a-f25015dc4e83","Type":"ContainerDied","Data":"efcbbddda0ca2a279f6d7c6f23e0d3d145334a146c12c8bec2f62bf423014e5b"}
Nov 02 18:29:08.825443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.825400    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:29:08.848396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.848367    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:29:08.919111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919029    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml\" (UniqueName: \"kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml\") pod \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") "
Nov 02 18:29:08.919111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume\") pod \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\" (UID: \"54e8c557-0b90-4592-b388-fc08b2fcc5b5\") "
Nov 02 18:29:08.919383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919365    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume\") pod \"01716d13-d310-4be8-8c2a-f25015dc4e83\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") "
Nov 02 18:29:08.919452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919402    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk\" (UniqueName: \"kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk\") pod \"01716d13-d310-4be8-8c2a-f25015dc4e83\" (UID: \"01716d13-d310-4be8-8c2a-f25015dc4e83\") "
Nov 02 18:29:08.919526 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919502    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume" pod "54e8c557-0b90-4592-b388-fc08b2fcc5b5" (UID: "54e8c557-0b90-4592-b388-fc08b2fcc5b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:08.919854 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919814    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume" pod "01716d13-d310-4be8-8c2a-f25015dc4e83" (UID: "01716d13-d310-4be8-8c2a-f25015dc4e83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:08.919995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.919977    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01716d13-d310-4be8-8c2a-f25015dc4e83-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:08.920060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.920001    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e8c557-0b90-4592-b388-fc08b2fcc5b5-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:08.921999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.921974    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml" pod "54e8c557-0b90-4592-b388-fc08b2fcc5b5" (UID: "54e8c557-0b90-4592-b388-fc08b2fcc5b5"). InnerVolumeSpecName "kube-api-access-j2fml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:08.922218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:08.922201    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk" pod "01716d13-d310-4be8-8c2a-f25015dc4e83" (UID: "01716d13-d310-4be8-8c2a-f25015dc4e83"). InnerVolumeSpecName "kube-api-access-cstxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:09.021201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.021168    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fml\" (UniqueName: \"kubernetes.io/projected/54e8c557-0b90-4592-b388-fc08b2fcc5b5-kube-api-access-j2fml\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:09.021487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.021462    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstxk\" (UniqueName: \"kubernetes.io/projected/01716d13-d310-4be8-8c2a-f25015dc4e83-kube-api-access-cstxk\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:09.688612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.688580    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"
Nov 02 18:29:09.689144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.688578    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k" event={"ID":"01716d13-d310-4be8-8c2a-f25015dc4e83","Type":"ContainerDied","Data":"f9c01ea50cf5a73a735c0a50f86ec2d67f125bccb86f4767c1da9f0bc00fe684"}
Nov 02 18:29:09.689144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.688734    5042 scope.go:122] "RemoveContainer" containerID="efcbbddda0ca2a279f6d7c6f23e0d3d145334a146c12c8bec2f62bf423014e5b"
Nov 02 18:29:09.691309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.691281    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59" event={"ID":"54e8c557-0b90-4592-b388-fc08b2fcc5b5","Type":"ContainerDied","Data":"65b9066b9e0fcc3c7731c9ec274c6f674bc7c17ff786a7bf23e45b9b605e47c7"}
Nov 02 18:29:09.691485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.691394    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"
Nov 02 18:29:09.704417 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.704388    5042 scope.go:122] "RemoveContainer" containerID="af0a9718ef882e647babae4b35065357c8e7411679390981fd7e45ad8ea2524a"
Nov 02 18:29:09.716228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.716184    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"]
Nov 02 18:29:09.721398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.721374    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-42d786-77876d7c85-sxz59"]
Nov 02 18:29:09.729626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.729592    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"]
Nov 02 18:29:09.733703 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:09.733669    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-42d786-cfc454fff-25p7k"]
Nov 02 18:29:11.545223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.545185    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" path="/var/lib/kubelet/pods/01716d13-d310-4be8-8c2a-f25015dc4e83/volumes"
Nov 02 18:29:11.545935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.545915    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" path="/var/lib/kubelet/pods/54e8c557-0b90-4592-b388-fc08b2fcc5b5/volumes"
Nov 02 18:29:11.548650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.548630    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"]
Nov 02 18:29:11.549256 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549243    5042 cpu_manager.go:414] "containerMap: removing container" podUID="530814f9-6689-4091-a006-a7f973b1395f" containerName="maas-api"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549257    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530814f9-6689-4091-a006-a7f973b1395f" containerName="maas-api"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549275    5042 cpu_manager.go:414] "containerMap: removing container" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerName="payload-processing"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549280    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerName="payload-processing"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549292    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549298    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549306    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy"
Nov 02 18:29:11.549313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549312    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549319    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ed73717a-e17c-4cb4-8b93-57e43a0bc919" containerName="payload-pre-processing"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549324    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed73717a-e17c-4cb4-8b93-57e43a0bc919" containerName="payload-pre-processing"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549333    5042 cpu_manager.go:414] "containerMap: removing container" podUID="afc6f1d9-87dc-454d-8951-55a5626905a0" containerName="maas-api"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549339    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afc6f1d9-87dc-454d-8951-55a5626905a0" containerName="maas-api"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549348    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c92574ad-87f5-408e-a4dc-1b1c46bc69a6" containerName="payload-processing"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549353    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92574ad-87f5-408e-a4dc-1b1c46bc69a6" containerName="payload-processing"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549367    5042 cpu_manager.go:414] "containerMap: removing container" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549371    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549380    5042 cpu_manager.go:414] "containerMap: removing container" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerName="payload-pre-processing"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549384    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerName="payload-pre-processing"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549397    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549402    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549417    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549431    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549442    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549446    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549456    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1566fab0-31d1-4ca4-bed5-60a5da3b5a71" containerName="revoke-keys"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549460    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1566fab0-31d1-4ca4-bed5-60a5da3b5a71" containerName="revoke-keys"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549468    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy"
Nov 02 18:29:11.549546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549472    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549562    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1159bd-e7f6-4ba4-9c5e-b1e66e174f78" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549570    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="01716d13-d310-4be8-8c2a-f25015dc4e83" containerName="payload-pre-processing"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549580    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549586    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca681ab-7126-4c85-9246-38815d12500f" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549593    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549600    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549606    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3249565-04a0-496f-92c1-afe90b3c5c90" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549612    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1566fab0-31d1-4ca4-bed5-60a5da3b5a71" containerName="revoke-keys"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549618    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb0f812c-97f1-4a4d-9ee3-9d135fd3b559" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549624    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="530814f9-6689-4091-a006-a7f973b1395f" containerName="maas-api"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549630    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e8c557-0b90-4592-b388-fc08b2fcc5b5" containerName="payload-processing"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549637    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549642    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed73717a-e17c-4cb4-8b93-57e43a0bc919" containerName="payload-pre-processing"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549648    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="afc6f1d9-87dc-454d-8951-55a5626905a0" containerName="maas-api"
Nov 02 18:29:11.550200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.549655    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92574ad-87f5-408e-a4dc-1b1c46bc69a6" containerName="payload-processing"
Nov 02 18:29:11.554996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.554973    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.558822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.558794    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:29:11.559018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.558905    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:29:11.564367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.564336    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"]
Nov 02 18:29:11.671636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.671597    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcdl\" (UniqueName: \"kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.671805 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.671734    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.672138 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.672084    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.773449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.773398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcdl\" (UniqueName: \"kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.773675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.773499    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.773675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.773659    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.774538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.774500    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.776964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.776937    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.782373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.782348    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcdl\" (UniqueName: \"kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl\") pod \"maas-api-revoke-sub-cabc5d2a-z5585\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:11.877639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:11.877532    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:12.169893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:12.169859    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"]
Nov 02 18:29:12.171288 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:29:12.171259    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278bdd67_726f_4833_9cc2_1134268339b5.slice/crio-e7bb222e48b438b290ab3acf1976a715aa738a6e200c5498681ba5e311632ed5 WatchSource:0}: Error finding container e7bb222e48b438b290ab3acf1976a715aa738a6e200c5498681ba5e311632ed5: Status 404 returned error can't find the container with id e7bb222e48b438b290ab3acf1976a715aa738a6e200c5498681ba5e311632ed5
Nov 02 18:29:12.760865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:12.760804    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585" event={"ID":"278bdd67-726f-4833-9cc2-1134268339b5","Type":"ContainerStarted","Data":"caaedea5e614e6b6509fc130bcb3603ede5c41b5341dfdec7fd56029e3f74b4e"}
Nov 02 18:29:12.761272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:12.760893    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585" event={"ID":"278bdd67-726f-4833-9cc2-1134268339b5","Type":"ContainerStarted","Data":"e7bb222e48b438b290ab3acf1976a715aa738a6e200c5498681ba5e311632ed5"}
Nov 02 18:29:12.783054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:12.782982    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585" podStartSLOduration=1.7829614120000001 podStartE2EDuration="1.782961412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:12.775148491 +0000 UTC m=+3365.354557891" watchObservedRunningTime="2026-11-02 18:29:12.782961412 +0000 UTC m=+3365.362370800"
Nov 02 18:29:13.770333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:13.770291    5042 generic.go:356] "Generic (PLEG): container finished" podID="278bdd67-726f-4833-9cc2-1134268339b5" containerID="caaedea5e614e6b6509fc130bcb3603ede5c41b5341dfdec7fd56029e3f74b4e" exitCode=0
Nov 02 18:29:13.770755 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:13.770377    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585" event={"ID":"278bdd67-726f-4833-9cc2-1134268339b5","Type":"ContainerDied","Data":"caaedea5e614e6b6509fc130bcb3603ede5c41b5341dfdec7fd56029e3f74b4e"}
Nov 02 18:29:14.967178 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:14.967151    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:15.025640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.025603    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl\" (UniqueName: \"kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl\") pod \"278bdd67-726f-4833-9cc2-1134268339b5\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") "
Nov 02 18:29:15.025879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.025676    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca\") pod \"278bdd67-726f-4833-9cc2-1134268339b5\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") "
Nov 02 18:29:15.025966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.025949    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca\") pod \"278bdd67-726f-4833-9cc2-1134268339b5\" (UID: \"278bdd67-726f-4833-9cc2-1134268339b5\") "
Nov 02 18:29:15.026374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.026345    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca" pod "278bdd67-726f-4833-9cc2-1134268339b5" (UID: "278bdd67-726f-4833-9cc2-1134268339b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:15.026713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.026686    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:15.028285 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.028256    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl" pod "278bdd67-726f-4833-9cc2-1134268339b5" (UID: "278bdd67-726f-4833-9cc2-1134268339b5"). InnerVolumeSpecName "kube-api-access-dgcdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:15.029129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.029105    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca" pod "278bdd67-726f-4833-9cc2-1134268339b5" (UID: "278bdd67-726f-4833-9cc2-1134268339b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:15.128453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.128408    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcdl\" (UniqueName: \"kubernetes.io/projected/278bdd67-726f-4833-9cc2-1134268339b5-kube-api-access-dgcdl\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:15.128615 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.128459    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/278bdd67-726f-4833-9cc2-1134268339b5-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:15.792574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.792536    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585" event={"ID":"278bdd67-726f-4833-9cc2-1134268339b5","Type":"ContainerDied","Data":"e7bb222e48b438b290ab3acf1976a715aa738a6e200c5498681ba5e311632ed5"}
Nov 02 18:29:15.792766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.792600    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7bb222e48b438b290ab3acf1976a715aa738a6e200c5498681ba5e311632ed5"
Nov 02 18:29:15.792766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:15.792550    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"
Nov 02 18:29:18.540622 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:29:18.540533    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:29:18.947230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:18.947136    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"]
Nov 02 18:29:21.850916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.850094    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"]
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.851816    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.851862    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07d1a73-10bb-4dfe-9a07-b5e81ccc6d83" containerName="istio-proxy"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.851882    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.851890    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.851944    5042 cpu_manager.go:414] "containerMap: removing container" podUID="278bdd67-726f-4833-9cc2-1134268339b5" containerName="revoke-keys"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.851954    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278bdd67-726f-4833-9cc2-1134268339b5" containerName="revoke-keys"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.852149    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2772b5c-8b7b-4e10-8a1c-4d2f3703c2b1" containerName="istio-proxy"
Nov 02 18:29:21.855125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.852160    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="278bdd67-726f-4833-9cc2-1134268339b5" containerName="revoke-keys"
Nov 02 18:29:21.862582 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.862552    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:21.862898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.862873    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"]
Nov 02 18:29:21.868539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.868512    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:29:21.868732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.868594    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:29:21.962468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.962434    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:21.962688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.962664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5tl\" (UniqueName: \"kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:21.962780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:21.962762    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.064596 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.064552    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5tl\" (UniqueName: \"kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.064784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.064654    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.064784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.064734    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.065332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.065310    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.067348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.067323    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.073152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.073129    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5tl\" (UniqueName: \"kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl\") pod \"maas-api-revoke-sub-08029aa8-bnt2t\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.183531 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.183450    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:22.474219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.474183    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"]
Nov 02 18:29:22.474431 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:29:22.474244    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dafb96_b522_4fd8_861e_b30c41402b88.slice/crio-20e52031b74d1ef02f69838acd6b94edf73956fa3d6705a093fc59aa4fcf2d5d WatchSource:0}: Error finding container 20e52031b74d1ef02f69838acd6b94edf73956fa3d6705a093fc59aa4fcf2d5d: Status 404 returned error can't find the container with id 20e52031b74d1ef02f69838acd6b94edf73956fa3d6705a093fc59aa4fcf2d5d
Nov 02 18:29:22.911365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:22.911324    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t" event={"ID":"b3dafb96-b522-4fd8-861e-b30c41402b88","Type":"ContainerStarted","Data":"20e52031b74d1ef02f69838acd6b94edf73956fa3d6705a093fc59aa4fcf2d5d"}
Nov 02 18:29:23.954926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:23.947774    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing" containerID="cri-o://0a15d77c590adada9a8b11739cb8e7c97a65b3307c00b7eb4d88b02bbf006610" gracePeriod=25
Nov 02 18:29:23.991343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:23.991295    5042 generic.go:356] "Generic (PLEG): container finished" podID="b3dafb96-b522-4fd8-861e-b30c41402b88" containerID="dcb5189f9da0eb7aaf1a1e6548efdf19e31ff5a5ba9c7037abf14186457f21f5" exitCode=0
Nov 02 18:29:23.991555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:23.991434    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t" event={"ID":"b3dafb96-b522-4fd8-861e-b30c41402b88","Type":"ContainerDied","Data":"dcb5189f9da0eb7aaf1a1e6548efdf19e31ff5a5ba9c7037abf14186457f21f5"}
Nov 02 18:29:24.552005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.551969    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"]
Nov 02 18:29:24.556533 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.556515    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.571986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.571955    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"]
Nov 02 18:29:24.609912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.609869    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.610202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.610178    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq8hj\" (UniqueName: \"kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.610305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.610287    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.712121 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.712079    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.712322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.712181    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.712385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.712334    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq8hj\" (UniqueName: \"kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.713037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.713010    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.714712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.714682    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.721026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.720997    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq8hj\" (UniqueName: \"kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj\") pod \"maas-api-revoke-sub-13a9cee9-hvvqq\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:24.873140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:24.873041    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:25.198884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.198802    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:25.205023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.204991    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"]
Nov 02 18:29:25.206589 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:29:25.206567    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201f4338_dc79_4142_b649_181ab2313ac8.slice/crio-c7e65a234107258abbb8c5bc3335f72e16390989347c9fdb9db6d2ed68afa2cf WatchSource:0}: Error finding container c7e65a234107258abbb8c5bc3335f72e16390989347c9fdb9db6d2ed68afa2cf: Status 404 returned error can't find the container with id c7e65a234107258abbb8c5bc3335f72e16390989347c9fdb9db6d2ed68afa2cf
Nov 02 18:29:25.333788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.333751    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca\") pod \"b3dafb96-b522-4fd8-861e-b30c41402b88\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") "
Nov 02 18:29:25.334030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.333945    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca\") pod \"b3dafb96-b522-4fd8-861e-b30c41402b88\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") "
Nov 02 18:29:25.334030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.334019    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl\" (UniqueName: \"kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl\") pod \"b3dafb96-b522-4fd8-861e-b30c41402b88\" (UID: \"b3dafb96-b522-4fd8-861e-b30c41402b88\") "
Nov 02 18:29:25.334295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.334270    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca" pod "b3dafb96-b522-4fd8-861e-b30c41402b88" (UID: "b3dafb96-b522-4fd8-861e-b30c41402b88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:25.334477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.334462    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:25.336261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.336237    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca" pod "b3dafb96-b522-4fd8-861e-b30c41402b88" (UID: "b3dafb96-b522-4fd8-861e-b30c41402b88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:25.336396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.336293    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl" pod "b3dafb96-b522-4fd8-861e-b30c41402b88" (UID: "b3dafb96-b522-4fd8-861e-b30c41402b88"). InnerVolumeSpecName "kube-api-access-fp5tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:25.435663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.435576    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp5tl\" (UniqueName: \"kubernetes.io/projected/b3dafb96-b522-4fd8-861e-b30c41402b88-kube-api-access-fp5tl\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:25.435663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:25.435608    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3dafb96-b522-4fd8-861e-b30c41402b88-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:26.024769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:26.024655    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq" event={"ID":"201f4338-dc79-4142-b649-181ab2313ac8","Type":"ContainerStarted","Data":"7244801d4e08418d5740d7891fbf2a04179c5996fd3a5322071cc8af503ccae9"}
Nov 02 18:29:26.024769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:26.024732    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq" event={"ID":"201f4338-dc79-4142-b649-181ab2313ac8","Type":"ContainerStarted","Data":"c7e65a234107258abbb8c5bc3335f72e16390989347c9fdb9db6d2ed68afa2cf"}
Nov 02 18:29:26.028315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:26.028288    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"
Nov 02 18:29:26.028315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:26.028302    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t" event={"ID":"b3dafb96-b522-4fd8-861e-b30c41402b88","Type":"ContainerDied","Data":"20e52031b74d1ef02f69838acd6b94edf73956fa3d6705a093fc59aa4fcf2d5d"}
Nov 02 18:29:26.028551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:26.028363    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e52031b74d1ef02f69838acd6b94edf73956fa3d6705a093fc59aa4fcf2d5d"
Nov 02 18:29:26.043775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:26.043720    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq" podStartSLOduration=2.043704887 podStartE2EDuration="2.043704887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:26.040455097 +0000 UTC m=+3378.619864490" watchObservedRunningTime="2026-11-02 18:29:26.043704887 +0000 UTC m=+3378.623114272"
Nov 02 18:29:27.023715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:27.023678    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"]
Nov 02 18:29:27.027958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:27.027929    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfeef8ff-mjxq9"]
Nov 02 18:29:27.037822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:27.037788    5042 generic.go:356] "Generic (PLEG): container finished" podID="201f4338-dc79-4142-b649-181ab2313ac8" containerID="7244801d4e08418d5740d7891fbf2a04179c5996fd3a5322071cc8af503ccae9" exitCode=0
Nov 02 18:29:27.038004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:27.037875    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq" event={"ID":"201f4338-dc79-4142-b649-181ab2313ac8","Type":"ContainerDied","Data":"7244801d4e08418d5740d7891fbf2a04179c5996fd3a5322071cc8af503ccae9"}
Nov 02 18:29:27.545376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:27.545344    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde" path="/var/lib/kubelet/pods/58ac6dd9-c41b-4bbd-bb7d-8ffc71448bde/volumes"
Nov 02 18:29:28.256257 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.256224    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:28.393484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.393379    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca\") pod \"201f4338-dc79-4142-b649-181ab2313ac8\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") "
Nov 02 18:29:28.393484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.393480    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca\") pod \"201f4338-dc79-4142-b649-181ab2313ac8\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") "
Nov 02 18:29:28.393691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.393553    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj\" (UniqueName: \"kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj\") pod \"201f4338-dc79-4142-b649-181ab2313ac8\" (UID: \"201f4338-dc79-4142-b649-181ab2313ac8\") "
Nov 02 18:29:28.393871 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.393816    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca" pod "201f4338-dc79-4142-b649-181ab2313ac8" (UID: "201f4338-dc79-4142-b649-181ab2313ac8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:28.394180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.394155    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:28.395791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.395768    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca" pod "201f4338-dc79-4142-b649-181ab2313ac8" (UID: "201f4338-dc79-4142-b649-181ab2313ac8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:28.396028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.396012    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj" pod "201f4338-dc79-4142-b649-181ab2313ac8" (UID: "201f4338-dc79-4142-b649-181ab2313ac8"). InnerVolumeSpecName "kube-api-access-vq8hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:28.452875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.452803    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"]
Nov 02 18:29:28.469083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.469046    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"]
Nov 02 18:29:28.506937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.506900    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/201f4338-dc79-4142-b649-181ab2313ac8-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:28.507219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:28.507194    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq8hj\" (UniqueName: \"kubernetes.io/projected/201f4338-dc79-4142-b649-181ab2313ac8-kube-api-access-vq8hj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:29.057447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:29.057402    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"
Nov 02 18:29:29.057657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:29.057422    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq" event={"ID":"201f4338-dc79-4142-b649-181ab2313ac8","Type":"ContainerDied","Data":"c7e65a234107258abbb8c5bc3335f72e16390989347c9fdb9db6d2ed68afa2cf"}
Nov 02 18:29:29.057657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:29.057544    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e65a234107258abbb8c5bc3335f72e16390989347c9fdb9db6d2ed68afa2cf"
Nov 02 18:29:29.261865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:29.260985    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"]
Nov 02 18:29:31.562661 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:31.562620    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:33.453627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:33.453567    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm" podUID="9536c99d-d79e-4177-b6a4-4d03ebab7a34" containerName="payload-processing" containerID="cri-o://ce0dd4042bab32ca91c943a01408ae4ee0eb5772a1a5fda896053928798187e0" gracePeriod=25
Nov 02 18:29:33.494680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:33.494630    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl" podUID="6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" containerName="payload-pre-processing" containerID="cri-o://503e4f225c896af2d2cc8f16441e1f9baa1a17341057378a7833808d1a55c3c5" gracePeriod=25
Nov 02 18:29:34.141500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.141189    5042 generic.go:356] "Generic (PLEG): container finished" podID="9536c99d-d79e-4177-b6a4-4d03ebab7a34" containerID="ce0dd4042bab32ca91c943a01408ae4ee0eb5772a1a5fda896053928798187e0" exitCode=0
Nov 02 18:29:34.141500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.141310    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm" event={"ID":"9536c99d-d79e-4177-b6a4-4d03ebab7a34","Type":"ContainerDied","Data":"ce0dd4042bab32ca91c943a01408ae4ee0eb5772a1a5fda896053928798187e0"}
Nov 02 18:29:34.150369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.150329    5042 generic.go:356] "Generic (PLEG): container finished" podID="6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" containerID="503e4f225c896af2d2cc8f16441e1f9baa1a17341057378a7833808d1a55c3c5" exitCode=0
Nov 02 18:29:34.150595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.150511    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl" event={"ID":"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a","Type":"ContainerDied","Data":"503e4f225c896af2d2cc8f16441e1f9baa1a17341057378a7833808d1a55c3c5"}
Nov 02 18:29:34.222878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.222057    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 18:29:34.252413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.252372    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp\" (UniqueName: \"kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp\") pod \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") "
Nov 02 18:29:34.252621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.252456    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume\") pod \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\" (UID: \"9536c99d-d79e-4177-b6a4-4d03ebab7a34\") "
Nov 02 18:29:34.253571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.253538    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume" pod "9536c99d-d79e-4177-b6a4-4d03ebab7a34" (UID: "9536c99d-d79e-4177-b6a4-4d03ebab7a34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:34.256427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.256399    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp" pod "9536c99d-d79e-4177-b6a4-4d03ebab7a34" (UID: "9536c99d-d79e-4177-b6a4-4d03ebab7a34"). InnerVolumeSpecName "kube-api-access-4ftrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:34.326272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.326242    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 18:29:34.356167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.353738    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume\") pod \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") "
Nov 02 18:29:34.356167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.353801    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll\" (UniqueName: \"kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll\") pod \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\" (UID: \"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a\") "
Nov 02 18:29:34.356167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.354169    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume" pod "6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" (UID: "6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:34.356167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.354216    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftrp\" (UniqueName: \"kubernetes.io/projected/9536c99d-d79e-4177-b6a4-4d03ebab7a34-kube-api-access-4ftrp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:34.356167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.354229    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9536c99d-d79e-4177-b6a4-4d03ebab7a34-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:34.358313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.358283    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll" pod "6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" (UID: "6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a"). InnerVolumeSpecName "kube-api-access-ctkll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:34.456161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.456120    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:34.456161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:34.456151    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctkll\" (UniqueName: \"kubernetes.io/projected/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a-kube-api-access-ctkll\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:35.025585 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.025538    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"]
Nov 02 18:29:35.026560 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026539    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" containerName="payload-pre-processing"
Nov 02 18:29:35.026614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026565    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" containerName="payload-pre-processing"
Nov 02 18:29:35.026614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026583    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b3dafb96-b522-4fd8-861e-b30c41402b88" containerName="revoke-keys"
Nov 02 18:29:35.026614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026593    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3dafb96-b522-4fd8-861e-b30c41402b88" containerName="revoke-keys"
Nov 02 18:29:35.026722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026633    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9536c99d-d79e-4177-b6a4-4d03ebab7a34" containerName="payload-processing"
Nov 02 18:29:35.026722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026641    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9536c99d-d79e-4177-b6a4-4d03ebab7a34" containerName="payload-processing"
Nov 02 18:29:35.026722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026669    5042 cpu_manager.go:414] "containerMap: removing container" podUID="201f4338-dc79-4142-b649-181ab2313ac8" containerName="revoke-keys"
Nov 02 18:29:35.026722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026677    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="201f4338-dc79-4142-b649-181ab2313ac8" containerName="revoke-keys"
Nov 02 18:29:35.026859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026813    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" containerName="payload-pre-processing"
Nov 02 18:29:35.026859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026825    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3dafb96-b522-4fd8-861e-b30c41402b88" containerName="revoke-keys"
Nov 02 18:29:35.026931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026858    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9536c99d-d79e-4177-b6a4-4d03ebab7a34" containerName="payload-processing"
Nov 02 18:29:35.026931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.026874    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="201f4338-dc79-4142-b649-181ab2313ac8" containerName="revoke-keys"
Nov 02 18:29:35.031400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.031371    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.035236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.035204    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"]
Nov 02 18:29:35.035452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.035230    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:29:35.036240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.036216    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:29:35.067027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.066986    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.067267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.067109    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.067349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.067267    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrtz\" (UniqueName: \"kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.160826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.160728    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"
Nov 02 18:29:35.160826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.160747    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm" event={"ID":"9536c99d-d79e-4177-b6a4-4d03ebab7a34","Type":"ContainerDied","Data":"c2fa104b000a740a5d40ea40f01c7421900ce28ae27c7e4cafe5cc0e58e7b7ab"}
Nov 02 18:29:35.160826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.160814    5042 scope.go:122] "RemoveContainer" containerID="ce0dd4042bab32ca91c943a01408ae4ee0eb5772a1a5fda896053928798187e0"
Nov 02 18:29:35.162612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.162574    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl" event={"ID":"6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a","Type":"ContainerDied","Data":"1c90bc6a2e8444e60d3ff17cc3a8b5225ae4c72d5d438f0ec2e7642d15240ee6"}
Nov 02 18:29:35.162764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.162683    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"
Nov 02 18:29:35.168356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.168325    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrtz\" (UniqueName: \"kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.168532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.168510    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.168620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.168599    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.169460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.169436    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.171538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.171518    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.178109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.178083    5042 scope.go:122] "RemoveContainer" containerID="503e4f225c896af2d2cc8f16441e1f9baa1a17341057378a7833808d1a55c3c5"
Nov 02 18:29:35.188764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.184061    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrtz\" (UniqueName: \"kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz\") pod \"maas-api-revoke-sub-31bc1d13-ccm8c\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.239792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.239684    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"]
Nov 02 18:29:35.248638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.248602    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-0f0baf-6f5bcfbd89-d2zdm"]
Nov 02 18:29:35.257969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.257919    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"]
Nov 02 18:29:35.267254 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.267196    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-0f0baf-5d5c9694d7-nr6nl"]
Nov 02 18:29:35.353662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.353621    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:35.648947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.648877    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a" path="/var/lib/kubelet/pods/6ef9de6c-1ea5-456d-a03f-2c4d1c02b36a/volumes"
Nov 02 18:29:35.650625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.650191    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9536c99d-d79e-4177-b6a4-4d03ebab7a34" path="/var/lib/kubelet/pods/9536c99d-d79e-4177-b6a4-4d03ebab7a34/volumes"
Nov 02 18:29:35.794467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.783309    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"]
Nov 02 18:29:35.978147 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.978104    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"]
Nov 02 18:29:35.978386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:35.978358    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" podUID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" containerName="istio-proxy" containerID="cri-o://4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd" gracePeriod=30
Nov 02 18:29:36.232276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:36.232239    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c" event={"ID":"3588267e-0d64-4fb2-be97-06b5776f28d4","Type":"ContainerStarted","Data":"55f4a5aa9962e80fa6fb0478c500e53b16bd507ba518b082dfe08a6e89ed56c1"}
Nov 02 18:29:37.249715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:37.249625    5042 generic.go:356] "Generic (PLEG): container finished" podID="3588267e-0d64-4fb2-be97-06b5776f28d4" containerID="f1a2a8a4beedf1aeadbe9db0fba1699305fbd5027f0a409e277b5061ddeb1db1" exitCode=0
Nov 02 18:29:37.250202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:37.249707    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c" event={"ID":"3588267e-0d64-4fb2-be97-06b5776f28d4","Type":"ContainerDied","Data":"f1a2a8a4beedf1aeadbe9db0fba1699305fbd5027f0a409e277b5061ddeb1db1"}
Nov 02 18:29:38.586120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.586032    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:38.696722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.696682    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz\" (UniqueName: \"kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz\") pod \"3588267e-0d64-4fb2-be97-06b5776f28d4\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") "
Nov 02 18:29:38.696969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.696752    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca\") pod \"3588267e-0d64-4fb2-be97-06b5776f28d4\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") "
Nov 02 18:29:38.696969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.696787    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca\") pod \"3588267e-0d64-4fb2-be97-06b5776f28d4\" (UID: \"3588267e-0d64-4fb2-be97-06b5776f28d4\") "
Nov 02 18:29:38.697542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.697507    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca" pod "3588267e-0d64-4fb2-be97-06b5776f28d4" (UID: "3588267e-0d64-4fb2-be97-06b5776f28d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:38.697951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.697924    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:38.700038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.700009    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca" pod "3588267e-0d64-4fb2-be97-06b5776f28d4" (UID: "3588267e-0d64-4fb2-be97-06b5776f28d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:38.700361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.700330    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz" pod "3588267e-0d64-4fb2-be97-06b5776f28d4" (UID: "3588267e-0d64-4fb2-be97-06b5776f28d4"). InnerVolumeSpecName "kube-api-access-2hrtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:38.799207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.799147    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrtz\" (UniqueName: \"kubernetes.io/projected/3588267e-0d64-4fb2-be97-06b5776f28d4-kube-api-access-2hrtz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:38.799207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:38.799183    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3588267e-0d64-4fb2-be97-06b5776f28d4-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:39.312895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:39.312831    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c" event={"ID":"3588267e-0d64-4fb2-be97-06b5776f28d4","Type":"ContainerDied","Data":"55f4a5aa9962e80fa6fb0478c500e53b16bd507ba518b082dfe08a6e89ed56c1"}
Nov 02 18:29:39.312895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:39.312827    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"
Nov 02 18:29:39.313176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:39.312919    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f4a5aa9962e80fa6fb0478c500e53b16bd507ba518b082dfe08a6e89ed56c1"
Nov 02 18:29:40.177001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.176919    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 18:29:40.225021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.224981    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225146    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225177    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225207    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225232    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225280    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225325    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225381    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.225503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.225432    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo\") pod \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\" (UID: \"c5fc31ec-cf6a-424d-8903-9a65607e3f27\") "
Nov 02 18:29:40.226638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.226099    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:40.226638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.226479    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:40.226638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.226548    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:40.226638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.226568    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:40.229269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.229226    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:40.229936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.229614    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:40.229936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.229802    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:40.229936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.229901    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "kube-api-access-nnwm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:40.230591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.230565    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo" pod "c5fc31ec-cf6a-424d-8903-9a65607e3f27" (UID: "c5fc31ec-cf6a-424d-8903-9a65607e3f27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:29:40.323471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.323440    5042 generic.go:356] "Generic (PLEG): container finished" podID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" containerID="4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd" exitCode=0
Nov 02 18:29:40.323666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.323492    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"
Nov 02 18:29:40.323666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.323532    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" event={"ID":"c5fc31ec-cf6a-424d-8903-9a65607e3f27","Type":"ContainerDied","Data":"4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd"}
Nov 02 18:29:40.323666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.323590    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698" event={"ID":"c5fc31ec-cf6a-424d-8903-9a65607e3f27","Type":"ContainerDied","Data":"b6437946179304cab6258a2b36972e78e111439fac7aa545761d581c925f4c81"}
Nov 02 18:29:40.323666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.323611    5042 scope.go:122] "RemoveContainer" containerID="4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd"
Nov 02 18:29:40.327292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327260    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327288    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnwm9\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-kube-api-access-nnwm9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327304    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327317    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327331    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327345    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327357    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5fc31ec-cf6a-424d-8903-9a65607e3f27-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327372    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.327468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.327385    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5fc31ec-cf6a-424d-8903-9a65607e3f27-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:40.336601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.336581    5042 scope.go:122] "RemoveContainer" containerID="4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd"
Nov 02 18:29:40.336936 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:29:40.336916    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd\": container with ID starting with 4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd not found: ID does not exist" containerID="4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd"
Nov 02 18:29:40.337011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.336946    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd"} err="failed to get container status \"4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd\": rpc error: code = NotFound desc = could not find container \"4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd\": container with ID starting with 4721ce86ce77d1bd20693835c3e274342a929fcd3b34dd303232c5ef1ea0b6cd not found: ID does not exist"
Nov 02 18:29:40.348267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.348242    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"]
Nov 02 18:29:40.354150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:40.354123    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0f0baf-openshift-default-66f587fff5-md698"]
Nov 02 18:29:41.544716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:41.544673    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" path="/var/lib/kubelet/pods/c5fc31ec-cf6a-424d-8903-9a65607e3f27/volumes"
Nov 02 18:29:41.562632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:41.562593    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:29:43.942758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:43.942719    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"]
Nov 02 18:29:43.955564 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:43.955528    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"]
Nov 02 18:29:44.376541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.376372    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" containerID="cri-o://a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148" gracePeriod=45
Nov 02 18:29:44.425416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.423780    5042 generic.go:356] "Generic (PLEG): container finished" podID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerID="0a15d77c590adada9a8b11739cb8e7c97a65b3307c00b7eb4d88b02bbf006610" exitCode=0
Nov 02 18:29:44.425416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.423876    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" event={"ID":"41b99141-2c4b-4e25-9d94-c8540619fe66","Type":"ContainerDied","Data":"0a15d77c590adada9a8b11739cb8e7c97a65b3307c00b7eb4d88b02bbf006610"}
Nov 02 18:29:44.562351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.562051    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:29:44.645879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.645805    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8\" (UniqueName: \"kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8\") pod \"41b99141-2c4b-4e25-9d94-c8540619fe66\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") "
Nov 02 18:29:44.646088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.646068    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume\") pod \"41b99141-2c4b-4e25-9d94-c8540619fe66\" (UID: \"41b99141-2c4b-4e25-9d94-c8540619fe66\") "
Nov 02 18:29:44.646904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.646877    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume" pod "41b99141-2c4b-4e25-9d94-c8540619fe66" (UID: "41b99141-2c4b-4e25-9d94-c8540619fe66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:44.650057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.650031    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8" pod "41b99141-2c4b-4e25-9d94-c8540619fe66" (UID: "41b99141-2c4b-4e25-9d94-c8540619fe66"). InnerVolumeSpecName "kube-api-access-h4nk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:44.748550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.748519    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nk8\" (UniqueName: \"kubernetes.io/projected/41b99141-2c4b-4e25-9d94-c8540619fe66-kube-api-access-h4nk8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.748550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.748553    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41b99141-2c4b-4e25-9d94-c8540619fe66-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.755092 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.755065    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:29:44.850024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.849988    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692\" (UniqueName: \"kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692\") pod \"72525843-3f51-443b-81d5-b146ebd118fb\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") "
Nov 02 18:29:44.850216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.850038    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm\") pod \"72525843-3f51-443b-81d5-b146ebd118fb\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") "
Nov 02 18:29:44.850216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.850168    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs\") pod \"72525843-3f51-443b-81d5-b146ebd118fb\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") "
Nov 02 18:29:44.850216 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.850206    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home\") pod \"72525843-3f51-443b-81d5-b146ebd118fb\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") "
Nov 02 18:29:44.850422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.850318    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir\") pod \"72525843-3f51-443b-81d5-b146ebd118fb\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") "
Nov 02 18:29:44.850422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.850412    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache\") pod \"72525843-3f51-443b-81d5-b146ebd118fb\" (UID: \"72525843-3f51-443b-81d5-b146ebd118fb\") "
Nov 02 18:29:44.851507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.851471    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home" pod "72525843-3f51-443b-81d5-b146ebd118fb" (UID: "72525843-3f51-443b-81d5-b146ebd118fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:44.851507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.851483    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir" pod "72525843-3f51-443b-81d5-b146ebd118fb" (UID: "72525843-3f51-443b-81d5-b146ebd118fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:44.851724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.851561    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache" pod "72525843-3f51-443b-81d5-b146ebd118fb" (UID: "72525843-3f51-443b-81d5-b146ebd118fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:44.852867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.852826    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692" pod "72525843-3f51-443b-81d5-b146ebd118fb" (UID: "72525843-3f51-443b-81d5-b146ebd118fb"). InnerVolumeSpecName "kube-api-access-fm692". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:44.852981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.852963    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm" pod "72525843-3f51-443b-81d5-b146ebd118fb" (UID: "72525843-3f51-443b-81d5-b146ebd118fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:29:44.852981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.852974    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs" pod "72525843-3f51-443b-81d5-b146ebd118fb" (UID: "72525843-3f51-443b-81d5-b146ebd118fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:44.962403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.954188    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72525843-3f51-443b-81d5-b146ebd118fb-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.962403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.954232    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.962403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.954246    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.962403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.954259    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.962403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.954274    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm692\" (UniqueName: \"kubernetes.io/projected/72525843-3f51-443b-81d5-b146ebd118fb-kube-api-access-fm692\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:44.962403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:44.954290    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72525843-3f51-443b-81d5-b146ebd118fb-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:45.435457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.435358    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"
Nov 02 18:29:45.435457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.435395    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr" event={"ID":"41b99141-2c4b-4e25-9d94-c8540619fe66","Type":"ContainerDied","Data":"ee20fa05e711bafd05081a91a337c16686fc7d4111b80884b049b19d2ef33157"}
Nov 02 18:29:45.435701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.435476    5042 scope.go:122] "RemoveContainer" containerID="0a15d77c590adada9a8b11739cb8e7c97a65b3307c00b7eb4d88b02bbf006610"
Nov 02 18:29:45.441766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.441423    5042 generic.go:356] "Generic (PLEG): container finished" podID="72525843-3f51-443b-81d5-b146ebd118fb" containerID="a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148" exitCode=0
Nov 02 18:29:45.441766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.441481    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"
Nov 02 18:29:45.441766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.441507    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" event={"ID":"72525843-3f51-443b-81d5-b146ebd118fb","Type":"ContainerDied","Data":"a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148"}
Nov 02 18:29:45.441766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.441569    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" event={"ID":"72525843-3f51-443b-81d5-b146ebd118fb","Type":"ContainerDied","Data":"2961f3aa55fa15192a4f692db62681799cef9777debd6e56b854737e44756f31"}
Nov 02 18:29:45.454218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.454196    5042 scope.go:122] "RemoveContainer" containerID="a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148"
Nov 02 18:29:45.464658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.464629    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"]
Nov 02 18:29:45.467397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.467375    5042 scope.go:122] "RemoveContainer" containerID="a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148"
Nov 02 18:29:45.467871 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:29:45.467812    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148\": container with ID starting with a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148 not found: ID does not exist" containerID="a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148"
Nov 02 18:29:45.468047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.468010    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148"} err="failed to get container status \"a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148\": rpc error: code = NotFound desc = could not find container \"a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148\": container with ID starting with a3ddb425e45b0ef3bafc6e7f4f1646ce69d0098810aba86ab83757e2a9332148 not found: ID does not exist"
Nov 02 18:29:45.469931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.469905    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a8e574-6866d758c-kfjgr"]
Nov 02 18:29:45.485172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.485141    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"]
Nov 02 18:29:45.496424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.496379    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6"]
Nov 02 18:29:45.545312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.545279    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" path="/var/lib/kubelet/pods/41b99141-2c4b-4e25-9d94-c8540619fe66/volumes"
Nov 02 18:29:45.545823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.545808    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72525843-3f51-443b-81d5-b146ebd118fb" path="/var/lib/kubelet/pods/72525843-3f51-443b-81d5-b146ebd118fb/volumes"
Nov 02 18:29:45.749876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:45.743047    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-24cfb6/premium-w1-24cfb6-kserve-644cbd56d4-rfvf6" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:29:48.539392 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:29:48.539352    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:29:48.944382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:48.944337    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6" podUID="1fe5753a-dd42-4a3b-8711-cf8868a0f61f" containerName="payload-processing" containerID="cri-o://43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7" gracePeriod=25
Nov 02 18:29:48.956439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:48.956394    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr" podUID="b82e6528-8aa6-48dc-8758-7d13c89902cf" containerName="payload-pre-processing" containerID="cri-o://cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594" gracePeriod=25
Nov 02 18:29:49.314110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.314082    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:29:49.353466 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.353435    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:29:49.362920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.362831    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume\") pod \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") "
Nov 02 18:29:49.363116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.362934    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs\" (UniqueName: \"kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs\") pod \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\" (UID: \"1fe5753a-dd42-4a3b-8711-cf8868a0f61f\") "
Nov 02 18:29:49.363116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.363005    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf\" (UniqueName: \"kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf\") pod \"b82e6528-8aa6-48dc-8758-7d13c89902cf\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") "
Nov 02 18:29:49.363116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.363032    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume\") pod \"b82e6528-8aa6-48dc-8758-7d13c89902cf\" (UID: \"b82e6528-8aa6-48dc-8758-7d13c89902cf\") "
Nov 02 18:29:49.363987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.363278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume" pod "1fe5753a-dd42-4a3b-8711-cf8868a0f61f" (UID: "1fe5753a-dd42-4a3b-8711-cf8868a0f61f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:49.363987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.363817    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume" pod "b82e6528-8aa6-48dc-8758-7d13c89902cf" (UID: "b82e6528-8aa6-48dc-8758-7d13c89902cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:49.363987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.363893    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:49.366100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.366068    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs" pod "1fe5753a-dd42-4a3b-8711-cf8868a0f61f" (UID: "1fe5753a-dd42-4a3b-8711-cf8868a0f61f"). InnerVolumeSpecName "kube-api-access-44sfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:49.366823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.366798    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf" pod "b82e6528-8aa6-48dc-8758-7d13c89902cf" (UID: "b82e6528-8aa6-48dc-8758-7d13c89902cf"). InnerVolumeSpecName "kube-api-access-tskxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:49.465446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.465342    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44sfs\" (UniqueName: \"kubernetes.io/projected/1fe5753a-dd42-4a3b-8711-cf8868a0f61f-kube-api-access-44sfs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:49.465446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.465382    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskxf\" (UniqueName: \"kubernetes.io/projected/b82e6528-8aa6-48dc-8758-7d13c89902cf-kube-api-access-tskxf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:49.465446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.465392    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b82e6528-8aa6-48dc-8758-7d13c89902cf-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:29:49.476602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.476568    5042 generic.go:356] "Generic (PLEG): container finished" podID="b82e6528-8aa6-48dc-8758-7d13c89902cf" containerID="cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594" exitCode=0
Nov 02 18:29:49.476787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.476626    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"
Nov 02 18:29:49.476787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.476659    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr" event={"ID":"b82e6528-8aa6-48dc-8758-7d13c89902cf","Type":"ContainerDied","Data":"cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594"}
Nov 02 18:29:49.476787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.476718    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr" event={"ID":"b82e6528-8aa6-48dc-8758-7d13c89902cf","Type":"ContainerDied","Data":"87ebc70e6f5d90abfb34cc7296917fcd306f44c711e1d783c91674e7394d33d6"}
Nov 02 18:29:49.476787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.476746    5042 scope.go:122] "RemoveContainer" containerID="cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594"
Nov 02 18:29:49.478820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.478785    5042 generic.go:356] "Generic (PLEG): container finished" podID="1fe5753a-dd42-4a3b-8711-cf8868a0f61f" containerID="43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7" exitCode=0
Nov 02 18:29:49.478975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.478826    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6" event={"ID":"1fe5753a-dd42-4a3b-8711-cf8868a0f61f","Type":"ContainerDied","Data":"43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7"}
Nov 02 18:29:49.478975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.478942    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6" event={"ID":"1fe5753a-dd42-4a3b-8711-cf8868a0f61f","Type":"ContainerDied","Data":"8a1782c06056db2f4bae2d6603a7e04bc014681d93fd7d94ae829b5949e8e0fd"}
Nov 02 18:29:49.478975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.478832    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"
Nov 02 18:29:49.490931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.490909    5042 scope.go:122] "RemoveContainer" containerID="cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594"
Nov 02 18:29:49.491454 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:29:49.491426    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594\": container with ID starting with cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594 not found: ID does not exist" containerID="cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594"
Nov 02 18:29:49.491557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.491465    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594"} err="failed to get container status \"cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594\": rpc error: code = NotFound desc = could not find container \"cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594\": container with ID starting with cd8f4c680d1c46a650ad546290acb99ffc868f023dfee45a9278191e577f6594 not found: ID does not exist"
Nov 02 18:29:49.491557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.491492    5042 scope.go:122] "RemoveContainer" containerID="43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7"
Nov 02 18:29:49.504600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.504577    5042 scope.go:122] "RemoveContainer" containerID="43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7"
Nov 02 18:29:49.505008 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:29:49.504982    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7\": container with ID starting with 43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7 not found: ID does not exist" containerID="43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7"
Nov 02 18:29:49.505089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.505021    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7"} err="failed to get container status \"43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7\": rpc error: code = NotFound desc = could not find container \"43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7\": container with ID starting with 43e2e16cc4b198a9ecfacc85e9ee93113fa40b5ce128563061fb7ab79fbb7ed7 not found: ID does not exist"
Nov 02 18:29:49.505489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.505454    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"]
Nov 02 18:29:49.509218 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.509193    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-72f605-6c67b54799-z6lxr"]
Nov 02 18:29:49.520550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.520519    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"]
Nov 02 18:29:49.525316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.525294    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-72f605-6bcd6bc985-gnbw6"]
Nov 02 18:29:49.544143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.544109    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe5753a-dd42-4a3b-8711-cf8868a0f61f" path="/var/lib/kubelet/pods/1fe5753a-dd42-4a3b-8711-cf8868a0f61f/volumes"
Nov 02 18:29:49.544630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:49.544613    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b82e6528-8aa6-48dc-8758-7d13c89902cf" path="/var/lib/kubelet/pods/b82e6528-8aa6-48dc-8758-7d13c89902cf/volumes"
Nov 02 18:29:52.262158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:52.262126    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"]
Nov 02 18:29:52.280288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:52.280251    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"]
Nov 02 18:29:57.263400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:57.263334    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerName="payload-processing" containerID="cri-o://d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8" gracePeriod=25
Nov 02 18:29:57.281165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:57.281112    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing" containerID="cri-o://f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085" gracePeriod=25
Nov 02 18:29:58.964134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:58.964100    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"]
Nov 02 18:29:58.964607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:58.964318    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy" containerID="cri-o://9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161" gracePeriod=30
Nov 02 18:29:59.027815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.027770    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"]
Nov 02 18:29:59.031345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.031316    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d68f851-w5cdw"]
Nov 02 18:29:59.544662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.544627    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a404b2b-754c-4c5b-8853-13afe0b0897a" path="/var/lib/kubelet/pods/5a404b2b-754c-4c5b-8853-13afe0b0897a/volumes"
Nov 02 18:29:59.803961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.803881    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"]
Nov 02 18:29:59.804492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804478    5042 cpu_manager.go:414] "containerMap: removing container" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main"
Nov 02 18:29:59.804546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804494    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main"
Nov 02 18:29:59.804546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804503    5042 cpu_manager.go:414] "containerMap: removing container" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing"
Nov 02 18:29:59.804546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804509    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing"
Nov 02 18:29:59.804546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804529    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3588267e-0d64-4fb2-be97-06b5776f28d4" containerName="revoke-keys"
Nov 02 18:29:59.804546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804534    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3588267e-0d64-4fb2-be97-06b5776f28d4" containerName="revoke-keys"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804551    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b82e6528-8aa6-48dc-8758-7d13c89902cf" containerName="payload-pre-processing"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804557    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b82e6528-8aa6-48dc-8758-7d13c89902cf" containerName="payload-pre-processing"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804580    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" containerName="istio-proxy"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804586    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" containerName="istio-proxy"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804594    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1fe5753a-dd42-4a3b-8711-cf8868a0f61f" containerName="payload-processing"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804598    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe5753a-dd42-4a3b-8711-cf8868a0f61f" containerName="payload-processing"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804684    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3588267e-0d64-4fb2-be97-06b5776f28d4" containerName="revoke-keys"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804694    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b82e6528-8aa6-48dc-8758-7d13c89902cf" containerName="payload-pre-processing"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804703    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fc31ec-cf6a-424d-8903-9a65607e3f27" containerName="istio-proxy"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804709    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe5753a-dd42-4a3b-8711-cf8868a0f61f" containerName="payload-processing"
Nov 02 18:29:59.804717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804719    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="72525843-3f51-443b-81d5-b146ebd118fb" containerName="main"
Nov 02 18:29:59.805654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.804725    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b99141-2c4b-4e25-9d94-c8540619fe66" containerName="payload-pre-processing"
Nov 02 18:29:59.810173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.810149    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:29:59.814324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.814291    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:29:59.814876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.814817    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:29:59.816342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.816316    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"]
Nov 02 18:29:59.923516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.923472    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k447j\" (UniqueName: \"kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:29:59.923716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.923682    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:29:59.923911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:29:59.923887    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.025120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.025072    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k447j\" (UniqueName: \"kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.025653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.025257    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.025653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.025406    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.026022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.025977    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.028303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.028279    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.035906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.035881    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k447j\" (UniqueName: \"kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j\") pod \"maas-api-revoke-sub-ac851bcf-f7smz\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.132570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.132460    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"]
Nov 02 18:30:00.137344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.137314    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.141057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.141022    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"]
Nov 02 18:30:00.146312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.146289    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:00.148821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.148774    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"]
Nov 02 18:30:00.152743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.151672    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.263316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.260132    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.263316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.260179    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sqzb\" (UniqueName: \"kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.263316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.260307    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.263316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.260339    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw586\" (UniqueName: \"kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.263316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.263254    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"]
Nov 02 18:30:00.269863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.269170    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.276434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.275339    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"]
Nov 02 18:30:00.276434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.275391    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"]
Nov 02 18:30:00.286865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.280304    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.286865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.282086    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"]
Nov 02 18:30:00.286865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.282125    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"]
Nov 02 18:30:00.286865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.282212    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.317262 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.315821    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"]
Nov 02 18:30:00.330396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.330368    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"]
Nov 02 18:30:00.331668 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.330966    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.343075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.343039    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"]
Nov 02 18:30:00.364917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.364885    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"]
Nov 02 18:30:00.374233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374124    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7h49\" (UniqueName: \"kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.374233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374204    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.374233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374230    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sqzb\" (UniqueName: \"kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.374452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374262    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.374452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374314    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.374452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374346    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzr7\" (UniqueName: \"kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.374452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374383    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.375083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374451    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkstn\" (UniqueName: \"kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.375083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374753    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.375083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374789    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw586\" (UniqueName: \"kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.375083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374860    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngng4\" (UniqueName: \"kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.375083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.374983    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.376488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.376457    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.377698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.377679    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"]
Nov 02 18:30:00.377978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.377961    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.379044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.379021    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.382346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.382323    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"]
Nov 02 18:30:00.401378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.395109    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sqzb\" (UniqueName: \"kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.401378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.401260    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw586\" (UniqueName: \"kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.458937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.458898    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"]
Nov 02 18:30:00.459453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.459420    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" containerID="cri-o://6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124" gracePeriod=30
Nov 02 18:30:00.476085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476052    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngng4\" (UniqueName: \"kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.476349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476332    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.476463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476444    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7h49\" (UniqueName: \"kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.476557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476517    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bmd\" (UniqueName: \"kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.476622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476555    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.476622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476606    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.476744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476636    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.476744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476674    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzr7\" (UniqueName: \"kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.476744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476719    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.476938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.476797    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkstn\" (UniqueName: \"kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.477296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.477268    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.477873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.477832    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.478007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.477988    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.478644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.478624    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.492475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.492441    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzr7\" (UniqueName: \"kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7\") pod \"maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.492644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.492441    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngng4\" (UniqueName: \"kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.494901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.494871    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkstn\" (UniqueName: \"kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.495431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.495406    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7h49\" (UniqueName: \"kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49\") pod \"maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.495545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.495463    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:00.528166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.528138    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"]
Nov 02 18:30:00.528299 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:00.528168    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0549c9_4136_492d_a11a_c98e27837baf.slice/crio-815b81e7062fa3c2ec502b60f976f08c34255dc2b2b24453a04f6abbd9751b55 WatchSource:0}: Error finding container 815b81e7062fa3c2ec502b60f976f08c34255dc2b2b24453a04f6abbd9751b55: Status 404 returned error can't find the container with id 815b81e7062fa3c2ec502b60f976f08c34255dc2b2b24453a04f6abbd9751b55
Nov 02 18:30:00.582920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.579014    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bmd\" (UniqueName: \"kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.582920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.579537    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.582920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.580315    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.590879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.588124    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bmd\" (UniqueName: \"kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.600454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.600417    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:00.779791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.761439    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:00.779791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.762132    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:00.779791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.768675    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:00.779791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.768882    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:00.779791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.768972    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:00.801813 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.801730    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz" event={"ID":"4f0549c9-4136-492d-a11a-c98e27837baf","Type":"ContainerStarted","Data":"815b81e7062fa3c2ec502b60f976f08c34255dc2b2b24453a04f6abbd9751b55"}
Nov 02 18:30:00.816254 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:00.816127    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"]
Nov 02 18:30:00.875176 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:00.875132    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13aa19a0_04cb_4da0_99dc_b4995196d1c2.slice/crio-1752b4a2f2ea4fd549f0d4e675c66d0e2eb49e9492b0f7f8a65b49faade83ecc WatchSource:0}: Error finding container 1752b4a2f2ea4fd549f0d4e675c66d0e2eb49e9492b0f7f8a65b49faade83ecc: Status 404 returned error can't find the container with id 1752b4a2f2ea4fd549f0d4e675c66d0e2eb49e9492b0f7f8a65b49faade83ecc
Nov 02 18:30:01.150220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.150194    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"]
Nov 02 18:30:01.181580 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:01.181523    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd837fe_25e5_4fbe_b677_3eb53110fa5e.slice/crio-4b6115ad095f971c7793a0f64aadfb567fa187b508bbdef36d768802a6e36bf3 WatchSource:0}: Error finding container 4b6115ad095f971c7793a0f64aadfb567fa187b508bbdef36d768802a6e36bf3: Status 404 returned error can't find the container with id 4b6115ad095f971c7793a0f64aadfb567fa187b508bbdef36d768802a6e36bf3
Nov 02 18:30:01.240759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.240731    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"]
Nov 02 18:30:01.269173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.269132    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"]
Nov 02 18:30:01.480116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.478758    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"]
Nov 02 18:30:01.480715 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:01.480676    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf15c7d0e_0668_454b_8c09_9ffa2745c28c.slice/crio-92f74336a1cf15f4f4f976902f9a39cbee5023dc9f97df9aec85534104592e7b WatchSource:0}: Error finding container 92f74336a1cf15f4f4f976902f9a39cbee5023dc9f97df9aec85534104592e7b: Status 404 returned error can't find the container with id 92f74336a1cf15f4f4f976902f9a39cbee5023dc9f97df9aec85534104592e7b
Nov 02 18:30:01.501437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.501352    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"]
Nov 02 18:30:01.507570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.507533    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"]
Nov 02 18:30:01.509593 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:01.509558    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99b18a5_b3a5_40a4_b968_68499c02e2d1.slice/crio-aeb9ed427da93bdf3ff8e9d2732a312db58783049e9a1ab10f4511deb560930c WatchSource:0}: Error finding container aeb9ed427da93bdf3ff8e9d2732a312db58783049e9a1ab10f4511deb560930c: Status 404 returned error can't find the container with id aeb9ed427da93bdf3ff8e9d2732a312db58783049e9a1ab10f4511deb560930c
Nov 02 18:30:01.510417 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:01.510394    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26ebda4_2401_4e89_9668_1f0f683b292a.slice/crio-94c038e569c914e5df1cb3bee3a5fe187d603b42f0db987d2954ee62ed19f8a3 WatchSource:0}: Error finding container 94c038e569c914e5df1cb3bee3a5fe187d603b42f0db987d2954ee62ed19f8a3: Status 404 returned error can't find the container with id 94c038e569c914e5df1cb3bee3a5fe187d603b42f0db987d2954ee62ed19f8a3
Nov 02 18:30:01.814033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.813995    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerStarted","Data":"2b57b6eaf0baa481c5e9a81df3cbf10c4b4540ce0f2a4cbd889dff3a01f680c0"}
Nov 02 18:30:01.816113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.816084    5042 generic.go:356] "Generic (PLEG): container finished" podID="4f0549c9-4136-492d-a11a-c98e27837baf" containerID="3a956d592af8193cc3328952016769c9d8f49f92873a7bb5cf068964fbc58763" exitCode=0
Nov 02 18:30:01.816265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.816171    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz" event={"ID":"4f0549c9-4136-492d-a11a-c98e27837baf","Type":"ContainerDied","Data":"3a956d592af8193cc3328952016769c9d8f49f92873a7bb5cf068964fbc58763"}
Nov 02 18:30:01.818041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.818013    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" event={"ID":"cba10682-eff3-45f2-91bf-33bb1df21c2e","Type":"ContainerStarted","Data":"bd58e4c915e39d6837e76b49bc5de47fc86510772d3fe96acbd3564198bb6ec1"}
Nov 02 18:30:01.820435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.820408    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerStarted","Data":"94c038e569c914e5df1cb3bee3a5fe187d603b42f0db987d2954ee62ed19f8a3"}
Nov 02 18:30:01.823922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.823827    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerStarted","Data":"7bc5d5f7f87284dc70d33740dedfdc8b31ae82d7d74a02aaee7b130d24754ad0"}
Nov 02 18:30:01.824124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.823933    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerStarted","Data":"4b6115ad095f971c7793a0f64aadfb567fa187b508bbdef36d768802a6e36bf3"}
Nov 02 18:30:01.825555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.825483    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" event={"ID":"b99b18a5-b3a5-40a4-b968-68499c02e2d1","Type":"ContainerStarted","Data":"aeb9ed427da93bdf3ff8e9d2732a312db58783049e9a1ab10f4511deb560930c"}
Nov 02 18:30:01.827206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.827178    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerStarted","Data":"92f74336a1cf15f4f4f976902f9a39cbee5023dc9f97df9aec85534104592e7b"}
Nov 02 18:30:01.829357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.829329    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerStarted","Data":"2c1a42ea62f82f847c91d2e116e50e5bb693534197c20569e19060aa8513e7c7"}
Nov 02 18:30:01.829487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.829380    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerStarted","Data":"1752b4a2f2ea4fd549f0d4e675c66d0e2eb49e9492b0f7f8a65b49faade83ecc"}
Nov 02 18:30:01.852472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.852412    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" podStartSLOduration=1.852396981 podStartE2EDuration="1.852396981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.848868078 +0000 UTC m=+3414.428277499" watchObservedRunningTime="2026-11-02 18:30:01.852396981 +0000 UTC m=+3414.431806373"
Nov 02 18:30:01.870865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:01.870773    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" podStartSLOduration=1.870754266 podStartE2EDuration="1.870754266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.866157869 +0000 UTC m=+3414.445567255" watchObservedRunningTime="2026-11-02 18:30:01.870754266 +0000 UTC m=+3414.450163655"
Nov 02 18:30:02.815909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.815863    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:02.816647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.816620    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:02.840745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.840709    5042 generic.go:356] "Generic (PLEG): container finished" podID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerID="c726a9b5b5bd0370d1896206e08d267255ceea951ef558702e15ce32048a4d20" exitCode=60
Nov 02 18:30:02.841002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.840801    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerDied","Data":"c726a9b5b5bd0370d1896206e08d267255ceea951ef558702e15ce32048a4d20"}
Nov 02 18:30:02.841225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.841200    5042 scope.go:122] "RemoveContainer" containerID="c726a9b5b5bd0370d1896206e08d267255ceea951ef558702e15ce32048a4d20"
Nov 02 18:30:02.843336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.843244    5042 generic.go:356] "Generic (PLEG): container finished" podID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerID="2c1a42ea62f82f847c91d2e116e50e5bb693534197c20569e19060aa8513e7c7" exitCode=60
Nov 02 18:30:02.843713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.843693    5042 scope.go:122] "RemoveContainer" containerID="2c1a42ea62f82f847c91d2e116e50e5bb693534197c20569e19060aa8513e7c7"
Nov 02 18:30:02.844108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.844027    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerDied","Data":"2c1a42ea62f82f847c91d2e116e50e5bb693534197c20569e19060aa8513e7c7"}
Nov 02 18:30:02.847016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.846575    5042 generic.go:356] "Generic (PLEG): container finished" podID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerID="c5e869ea130168f5beb8d742169307ca88c101edd4b406b0cd9be7b0dad515a3" exitCode=60
Nov 02 18:30:02.847016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.846675    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerDied","Data":"c5e869ea130168f5beb8d742169307ca88c101edd4b406b0cd9be7b0dad515a3"}
Nov 02 18:30:02.847016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.846912    5042 scope.go:122] "RemoveContainer" containerID="c5e869ea130168f5beb8d742169307ca88c101edd4b406b0cd9be7b0dad515a3"
Nov 02 18:30:02.854043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.854011    5042 generic.go:356] "Generic (PLEG): container finished" podID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerID="6026ab0b38f61b5d38fa903b36cd285351bcd311c42893a29af1c142725bd7cd" exitCode=60
Nov 02 18:30:02.854183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.854057    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" event={"ID":"cba10682-eff3-45f2-91bf-33bb1df21c2e","Type":"ContainerDied","Data":"6026ab0b38f61b5d38fa903b36cd285351bcd311c42893a29af1c142725bd7cd"}
Nov 02 18:30:02.854473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.854456    5042 scope.go:122] "RemoveContainer" containerID="6026ab0b38f61b5d38fa903b36cd285351bcd311c42893a29af1c142725bd7cd"
Nov 02 18:30:02.857692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.857632    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerStarted","Data":"616fb9073c27209ba6ea191556c659c557589d6468d3c61b0c852ee9fbb51ef6"}
Nov 02 18:30:02.860759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.860732    5042 generic.go:356] "Generic (PLEG): container finished" podID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerID="7bc5d5f7f87284dc70d33740dedfdc8b31ae82d7d74a02aaee7b130d24754ad0" exitCode=60
Nov 02 18:30:02.861049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.861028    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerDied","Data":"7bc5d5f7f87284dc70d33740dedfdc8b31ae82d7d74a02aaee7b130d24754ad0"}
Nov 02 18:30:02.861183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.861164    5042 scope.go:122] "RemoveContainer" containerID="7bc5d5f7f87284dc70d33740dedfdc8b31ae82d7d74a02aaee7b130d24754ad0"
Nov 02 18:30:02.862973 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.862950    5042 generic.go:356] "Generic (PLEG): container finished" podID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerID="d1495f844adf61173361f766041ab860d88c4e47f34f1e8724046ce67ed28f69" exitCode=60
Nov 02 18:30:02.863258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.863063    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" event={"ID":"b99b18a5-b3a5-40a4-b968-68499c02e2d1","Type":"ContainerDied","Data":"d1495f844adf61173361f766041ab860d88c4e47f34f1e8724046ce67ed28f69"}
Nov 02 18:30:02.863323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.863251    5042 scope.go:122] "RemoveContainer" containerID="d1495f844adf61173361f766041ab860d88c4e47f34f1e8724046ce67ed28f69"
Nov 02 18:30:02.970916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:02.970728    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" podStartSLOduration=2.970707071 podStartE2EDuration="2.970707071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:02.947172688 +0000 UTC m=+3415.526582075" watchObservedRunningTime="2026-11-02 18:30:02.970707071 +0000 UTC m=+3415.550116459"
Nov 02 18:30:03.230595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.230564    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:03.337559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.337421    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j\" (UniqueName: \"kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j\") pod \"4f0549c9-4136-492d-a11a-c98e27837baf\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") "
Nov 02 18:30:03.337776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.337568    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca\") pod \"4f0549c9-4136-492d-a11a-c98e27837baf\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") "
Nov 02 18:30:03.337776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.337673    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca\") pod \"4f0549c9-4136-492d-a11a-c98e27837baf\" (UID: \"4f0549c9-4136-492d-a11a-c98e27837baf\") "
Nov 02 18:30:03.338193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.338163    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca" pod "4f0549c9-4136-492d-a11a-c98e27837baf" (UID: "4f0549c9-4136-492d-a11a-c98e27837baf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:03.338507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.338463    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:03.373390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.373338    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j" pod "4f0549c9-4136-492d-a11a-c98e27837baf" (UID: "4f0549c9-4136-492d-a11a-c98e27837baf"). InnerVolumeSpecName "kube-api-access-k447j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:03.377549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.377506    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca" pod "4f0549c9-4136-492d-a11a-c98e27837baf" (UID: "4f0549c9-4136-492d-a11a-c98e27837baf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:03.439450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.439420    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k447j\" (UniqueName: \"kubernetes.io/projected/4f0549c9-4136-492d-a11a-c98e27837baf-kube-api-access-k447j\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:03.439450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.439450    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0549c9-4136-492d-a11a-c98e27837baf-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:03.485336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.485301    5042 patch_prober.go:29] interesting pod/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:30:03.485529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.485361    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:30:03.880955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.880474    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerStarted","Data":"a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56"}
Nov 02 18:30:03.885789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.885749    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerStarted","Data":"92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00"}
Nov 02 18:30:03.888247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.888209    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerStarted","Data":"197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b"}
Nov 02 18:30:03.890098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.890076    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"
Nov 02 18:30:03.890236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.890098    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz" event={"ID":"4f0549c9-4136-492d-a11a-c98e27837baf","Type":"ContainerDied","Data":"815b81e7062fa3c2ec502b60f976f08c34255dc2b2b24453a04f6abbd9751b55"}
Nov 02 18:30:03.890236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.890148    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815b81e7062fa3c2ec502b60f976f08c34255dc2b2b24453a04f6abbd9751b55"
Nov 02 18:30:03.892525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.892494    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" event={"ID":"cba10682-eff3-45f2-91bf-33bb1df21c2e","Type":"ContainerStarted","Data":"270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf"}
Nov 02 18:30:03.894769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.894746    5042 generic.go:356] "Generic (PLEG): container finished" podID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerID="616fb9073c27209ba6ea191556c659c557589d6468d3c61b0c852ee9fbb51ef6" exitCode=60
Nov 02 18:30:03.895280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.894822    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerDied","Data":"616fb9073c27209ba6ea191556c659c557589d6468d3c61b0c852ee9fbb51ef6"}
Nov 02 18:30:03.895280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.895158    5042 scope.go:122] "RemoveContainer" containerID="616fb9073c27209ba6ea191556c659c557589d6468d3c61b0c852ee9fbb51ef6"
Nov 02 18:30:03.898147 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.898121    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerStarted","Data":"42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3"}
Nov 02 18:30:03.900882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.900814    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" event={"ID":"b99b18a5-b3a5-40a4-b968-68499c02e2d1","Type":"ContainerStarted","Data":"8218f70a6f57a52b5465c57021cdb6d6cfe355f8c2752f58e924bfd4a01df3e9"}
Nov 02 18:30:03.903250 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.903202    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" podStartSLOduration=3.903187135 podStartE2EDuration="3.903187135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.896981256 +0000 UTC m=+3416.476390642" watchObservedRunningTime="2026-11-02 18:30:03.903187135 +0000 UTC m=+3416.482596522"
Nov 02 18:30:03.915170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.915121    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" podStartSLOduration=3.915104914 podStartE2EDuration="3.915104914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.912800472 +0000 UTC m=+3416.492209858" watchObservedRunningTime="2026-11-02 18:30:03.915104914 +0000 UTC m=+3416.494514359"
Nov 02 18:30:03.936934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.936882    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" podStartSLOduration=3.936866228 podStartE2EDuration="3.936866228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.933096292 +0000 UTC m=+3416.512505676" watchObservedRunningTime="2026-11-02 18:30:03.936866228 +0000 UTC m=+3416.516275613"
Nov 02 18:30:03.980990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:03.980929    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" podStartSLOduration=3.980907933 podStartE2EDuration="3.980907933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.979072371 +0000 UTC m=+3416.558481757" watchObservedRunningTime="2026-11-02 18:30:03.980907933 +0000 UTC m=+3416.560317324"
Nov 02 18:30:04.412240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.412199    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"]
Nov 02 18:30:04.504722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.504694    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:30:04.556173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556126    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556173 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556178    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556229    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556248    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556334    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556479    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556511    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556578    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.556623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.556614    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket\") pod \"c04ba0c7-e939-4825-81be-080c7b4139d9\" (UID: \"c04ba0c7-e939-4825-81be-080c7b4139d9\") "
Nov 02 18:30:04.557332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.557121    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:04.557332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.557265    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:04.557332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.557318    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:04.557595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.557405    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:04.557595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.557469    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:04.559737 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.559708    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:04.559900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.559832    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:30:04.560125 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.560084    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:04.560418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.560398    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4" pod "c04ba0c7-e939-4825-81be-080c7b4139d9" (UID: "c04ba0c7-e939-4825-81be-080c7b4139d9"). InnerVolumeSpecName "kube-api-access-m25j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:04.658580 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658543    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.658580 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658580    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658596    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658614    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658628    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25j4\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-kube-api-access-m25j4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658643    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658658    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04ba0c7-e939-4825-81be-080c7b4139d9-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658673    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04ba0c7-e939-4825-81be-080c7b4139d9-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.659323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.658688    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04ba0c7-e939-4825-81be-080c7b4139d9-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:04.913361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.913314    5042 generic.go:356] "Generic (PLEG): container finished" podID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerID="197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b" exitCode=60
Nov 02 18:30:04.913927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.913407    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerDied","Data":"197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b"}
Nov 02 18:30:04.913927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.913515    5042 scope.go:122] "RemoveContainer" containerID="c5e869ea130168f5beb8d742169307ca88c101edd4b406b0cd9be7b0dad515a3"
Nov 02 18:30:04.914065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.913936    5042 scope.go:122] "RemoveContainer" containerID="197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b"
Nov 02 18:30:04.914366 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:04.914336    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk_odh-ai-gateway-infra(dbd96234-118e-41f0-aa6b-6c778e49fcc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4"
Nov 02 18:30:04.915796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.915769    5042 generic.go:356] "Generic (PLEG): container finished" podID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerID="270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf" exitCode=60
Nov 02 18:30:04.916015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.915870    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" event={"ID":"cba10682-eff3-45f2-91bf-33bb1df21c2e","Type":"ContainerDied","Data":"270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf"}
Nov 02 18:30:04.916249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.916222    5042 scope.go:122] "RemoveContainer" containerID="270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf"
Nov 02 18:30:04.916633 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:04.916597    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx_odh-ai-gateway-infra(cba10682-eff3-45f2-91bf-33bb1df21c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e"
Nov 02 18:30:04.918261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.918224    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerStarted","Data":"f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f"}
Nov 02 18:30:04.920456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.920433    5042 generic.go:356] "Generic (PLEG): container finished" podID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerID="42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3" exitCode=60
Nov 02 18:30:04.920588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.920525    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerDied","Data":"42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3"}
Nov 02 18:30:04.920895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.920829    5042 scope.go:122] "RemoveContainer" containerID="42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3"
Nov 02 18:30:04.921282 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:04.921259    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z_odh-ai-gateway-infra(0dd837fe-25e5-4fbe-b677-3eb53110fa5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e"
Nov 02 18:30:04.922719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.922693    5042 generic.go:356] "Generic (PLEG): container finished" podID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerID="8218f70a6f57a52b5465c57021cdb6d6cfe355f8c2752f58e924bfd4a01df3e9" exitCode=60
Nov 02 18:30:04.922856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.922797    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" event={"ID":"b99b18a5-b3a5-40a4-b968-68499c02e2d1","Type":"ContainerDied","Data":"8218f70a6f57a52b5465c57021cdb6d6cfe355f8c2752f58e924bfd4a01df3e9"}
Nov 02 18:30:04.923065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.923047    5042 scope.go:122] "RemoveContainer" containerID="8218f70a6f57a52b5465c57021cdb6d6cfe355f8c2752f58e924bfd4a01df3e9"
Nov 02 18:30:04.923377 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:04.923356    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt_odh-ai-gateway-infra(b99b18a5-b3a5-40a4-b968-68499c02e2d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1"
Nov 02 18:30:04.924909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.924885    5042 generic.go:356] "Generic (PLEG): container finished" podID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerID="9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161" exitCode=0
Nov 02 18:30:04.925013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.924928    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"
Nov 02 18:30:04.925013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.924931    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" event={"ID":"c04ba0c7-e939-4825-81be-080c7b4139d9","Type":"ContainerDied","Data":"9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161"}
Nov 02 18:30:04.925013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.924974    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn" event={"ID":"c04ba0c7-e939-4825-81be-080c7b4139d9","Type":"ContainerDied","Data":"15aea4b72350e16665c16842149d97d0b05e53021c802f2f7a91c3076f26a911"}
Nov 02 18:30:04.927118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.927096    5042 generic.go:356] "Generic (PLEG): container finished" podID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerID="a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56" exitCode=60
Nov 02 18:30:04.927248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.927125    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerDied","Data":"a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56"}
Nov 02 18:30:04.927444 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.927428    5042 scope.go:122] "RemoveContainer" containerID="a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56"
Nov 02 18:30:04.927744 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:04.927723    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k_odh-ai-gateway-infra(f15c7d0e-0668-454b-8c09-9ffa2745c28c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c"
Nov 02 18:30:04.930227 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.930202    5042 generic.go:356] "Generic (PLEG): container finished" podID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerID="92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00" exitCode=60
Nov 02 18:30:04.930350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.930224    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerDied","Data":"92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00"}
Nov 02 18:30:04.930577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.930555    5042 scope.go:122] "RemoveContainer" containerID="92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00"
Nov 02 18:30:04.930930 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:04.930905    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g_odh-ai-gateway-infra(13aa19a0-04cb-4da0-99dc-b4995196d1c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2"
Nov 02 18:30:04.932573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.932549    5042 scope.go:122] "RemoveContainer" containerID="6026ab0b38f61b5d38fa903b36cd285351bcd311c42893a29af1c142725bd7cd"
Nov 02 18:30:04.950766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.950740    5042 scope.go:122] "RemoveContainer" containerID="7bc5d5f7f87284dc70d33740dedfdc8b31ae82d7d74a02aaee7b130d24754ad0"
Nov 02 18:30:04.969054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:04.969031    5042 scope.go:122] "RemoveContainer" containerID="d1495f844adf61173361f766041ab860d88c4e47f34f1e8724046ce67ed28f69"
Nov 02 18:30:05.029056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.028981    5042 scope.go:122] "RemoveContainer" containerID="9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161"
Nov 02 18:30:05.035424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.035392    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"]
Nov 02 18:30:05.039757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.039731    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-72f605-openshift-default-5bd6c4dc87-jzmwn"]
Nov 02 18:30:05.043449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.043424    5042 scope.go:122] "RemoveContainer" containerID="0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"
Nov 02 18:30:05.056595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.056573    5042 scope.go:122] "RemoveContainer" containerID="9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161"
Nov 02 18:30:05.056973 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.056947    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161\": container with ID starting with 9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161 not found: ID does not exist" containerID="9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161"
Nov 02 18:30:05.057076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.056981    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161"} err="failed to get container status \"9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161\": rpc error: code = NotFound desc = could not find container \"9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161\": container with ID starting with 9203c4144471ebad502dd26914eacc58329d34d5a4042da80687c56544c13161 not found: ID does not exist"
Nov 02 18:30:05.057076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.057001    5042 scope.go:122] "RemoveContainer" containerID="0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"
Nov 02 18:30:05.057264 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.057241    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38\": container with ID starting with 0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38 not found: ID does not exist" containerID="0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"
Nov 02 18:30:05.057348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.057276    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38"} err="failed to get container status \"0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38\": rpc error: code = NotFound desc = could not find container \"0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38\": container with ID starting with 0c10fe832dbcf18b212ad56cf3820b7ab96c4750b07c7c251dc41271b3a5fa38 not found: ID does not exist"
Nov 02 18:30:05.057348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.057298    5042 scope.go:122] "RemoveContainer" containerID="c726a9b5b5bd0370d1896206e08d267255ceea951ef558702e15ce32048a4d20"
Nov 02 18:30:05.090075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.090048    5042 scope.go:122] "RemoveContainer" containerID="2c1a42ea62f82f847c91d2e116e50e5bb693534197c20569e19060aa8513e7c7"
Nov 02 18:30:05.559506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.559463    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" path="/var/lib/kubelet/pods/c04ba0c7-e939-4825-81be-080c7b4139d9/volumes"
Nov 02 18:30:05.775156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.775129    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:30:05.878552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878516    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878576    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878613    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878638    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878678    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878742    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878866    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878901    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.878992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.878932    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy\") pod \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\" (UID: \"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6\") "
Nov 02 18:30:05.879534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.879502    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:05.879678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.879580    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:05.879746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.879731    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:05.879983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.879690    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:05.880227 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.880205    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:05.882436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.882408    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:05.882548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.882465    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "kube-api-access-7qvp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:05.882548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.882476    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:05.882548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.882487    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo" pod "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" (UID: "9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:30:05.942993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.942958    5042 scope.go:122] "RemoveContainer" containerID="a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56"
Nov 02 18:30:05.943491 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.943317    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k_odh-ai-gateway-infra(f15c7d0e-0668-454b-8c09-9ffa2745c28c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c"
Nov 02 18:30:05.945018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.944993    5042 scope.go:122] "RemoveContainer" containerID="92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00"
Nov 02 18:30:05.945371 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.945346    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g_odh-ai-gateway-infra(13aa19a0-04cb-4da0-99dc-b4995196d1c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2"
Nov 02 18:30:05.946749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.946728    5042 scope.go:122] "RemoveContainer" containerID="197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b"
Nov 02 18:30:05.947169 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.947136    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk_odh-ai-gateway-infra(dbd96234-118e-41f0-aa6b-6c778e49fcc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4"
Nov 02 18:30:05.948437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.948417    5042 scope.go:122] "RemoveContainer" containerID="270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf"
Nov 02 18:30:05.948723 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.948692    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx_odh-ai-gateway-infra(cba10682-eff3-45f2-91bf-33bb1df21c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e"
Nov 02 18:30:05.950393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.950369    5042 generic.go:356] "Generic (PLEG): container finished" podID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerID="f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f" exitCode=60
Nov 02 18:30:05.950493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.950404    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerDied","Data":"f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f"}
Nov 02 18:30:05.950493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.950456    5042 scope.go:122] "RemoveContainer" containerID="616fb9073c27209ba6ea191556c659c557589d6468d3c61b0c852ee9fbb51ef6"
Nov 02 18:30:05.950764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.950744    5042 scope.go:122] "RemoveContainer" containerID="f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f"
Nov 02 18:30:05.951099 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.951062    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr_odh-ai-gateway-infra(b26ebda4-2401-4e89-9668-1f0f683b292a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a"
Nov 02 18:30:05.952807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.952790    5042 scope.go:122] "RemoveContainer" containerID="42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3"
Nov 02 18:30:05.953061 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.953040    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z_odh-ai-gateway-infra(0dd837fe-25e5-4fbe-b677-3eb53110fa5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e"
Nov 02 18:30:05.954641 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.954623    5042 scope.go:122] "RemoveContainer" containerID="8218f70a6f57a52b5465c57021cdb6d6cfe355f8c2752f58e924bfd4a01df3e9"
Nov 02 18:30:05.954930 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:05.954905    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt_odh-ai-gateway-infra(b99b18a5-b3a5-40a4-b968-68499c02e2d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1"
Nov 02 18:30:05.956618 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.956596    5042 generic.go:356] "Generic (PLEG): container finished" podID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerID="6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124" exitCode=0
Nov 02 18:30:05.956733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.956631    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" event={"ID":"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6","Type":"ContainerDied","Data":"6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124"}
Nov 02 18:30:05.956733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.956647    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"
Nov 02 18:30:05.956733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.956676    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c" event={"ID":"9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6","Type":"ContainerDied","Data":"673eed2ef67a8a7d125586af8b2fc0889ac708d988cd46bb343d52cc0443d715"}
Nov 02 18:30:05.968964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.968942    5042 scope.go:122] "RemoveContainer" containerID="6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124"
Nov 02 18:30:05.982037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982004    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982040    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvp4\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-kube-api-access-7qvp4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982055    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982068    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982084    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982098    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982112    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982125    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.982264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.982178    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:05.984307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:05.984286    5042 scope.go:122] "RemoveContainer" containerID="d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"
Nov 02 18:30:06.002853 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.002813    5042 scope.go:122] "RemoveContainer" containerID="6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124"
Nov 02 18:30:06.003304 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:06.003271    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124\": container with ID starting with 6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124 not found: ID does not exist" containerID="6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124"
Nov 02 18:30:06.003414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.003315    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124"} err="failed to get container status \"6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124\": rpc error: code = NotFound desc = could not find container \"6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124\": container with ID starting with 6d388547707eff49c024f1395afca5ab71bc6fcf22a4791d89e576df1ab44124 not found: ID does not exist"
Nov 02 18:30:06.003414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.003340    5042 scope.go:122] "RemoveContainer" containerID="d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"
Nov 02 18:30:06.003687 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:06.003658    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234\": container with ID starting with d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234 not found: ID does not exist" containerID="d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"
Nov 02 18:30:06.003763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.003697    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234"} err="failed to get container status \"d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234\": rpc error: code = NotFound desc = could not find container \"d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234\": container with ID starting with d25b6605fc4ec98b7ae8135d2f3c82bc6e353ec948c1600169f6dfb9a472b234 not found: ID does not exist"
Nov 02 18:30:06.087535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.087500    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"]
Nov 02 18:30:06.094067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.094033    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c3c1f-openshift-default-68b7fcf5d9-9nb2c"]
Nov 02 18:30:06.969321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:06.969289    5042 scope.go:122] "RemoveContainer" containerID="f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f"
Nov 02 18:30:06.969789 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:06.969693    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr_odh-ai-gateway-infra(b26ebda4-2401-4e89-9668-1f0f683b292a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a"
Nov 02 18:30:07.616736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:07.616693    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" path="/var/lib/kubelet/pods/9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6/volumes"
Nov 02 18:30:08.525654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.525616    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"]
Nov 02 18:30:08.888865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.883077    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:08.974640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.974593    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn\" (UniqueName: \"kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn\") pod \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") "
Nov 02 18:30:08.974877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.974777    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca\") pod \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\" (UID: \"b99b18a5-b3a5-40a4-b968-68499c02e2d1\") "
Nov 02 18:30:08.975268 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.975240    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca" pod "b99b18a5-b3a5-40a4-b968-68499c02e2d1" (UID: "b99b18a5-b3a5-40a4-b968-68499c02e2d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:08.975445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.975426    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b99b18a5-b3a5-40a4-b968-68499c02e2d1-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:08.977954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.977924    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn" pod "b99b18a5-b3a5-40a4-b968-68499c02e2d1" (UID: "b99b18a5-b3a5-40a4-b968-68499c02e2d1"). InnerVolumeSpecName "kube-api-access-wkstn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:08.990373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.990337    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt" event={"ID":"b99b18a5-b3a5-40a4-b968-68499c02e2d1","Type":"ContainerDied","Data":"aeb9ed427da93bdf3ff8e9d2732a312db58783049e9a1ab10f4511deb560930c"}
Nov 02 18:30:08.990496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.990414    5042 scope.go:122] "RemoveContainer" containerID="8218f70a6f57a52b5465c57021cdb6d6cfe355f8c2752f58e924bfd4a01df3e9"
Nov 02 18:30:08.990496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:08.990353    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"
Nov 02 18:30:09.032819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:09.032787    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"]
Nov 02 18:30:09.036794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:09.036757    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-86f5f0-29894070-m68jt"]
Nov 02 18:30:09.077149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:09.077054    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkstn\" (UniqueName: \"kubernetes.io/projected/b99b18a5-b3a5-40a4-b968-68499c02e2d1-kube-api-access-wkstn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:09.545401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:09.545334    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" path="/var/lib/kubelet/pods/b99b18a5-b3a5-40a4-b968-68499c02e2d1/volumes"
Nov 02 18:30:11.028629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:11.028583    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"]
Nov 02 18:30:11.032851 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:11.032802    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b307226f-jhqw8"]
Nov 02 18:30:11.544346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:11.544302    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c729eb-e471-489e-b647-77062c55a25c" path="/var/lib/kubelet/pods/33c729eb-e471-489e-b647-77062c55a25c/volumes"
Nov 02 18:30:12.141305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141269    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"]
Nov 02 18:30:12.141948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141931    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy"
Nov 02 18:30:12.142008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141951    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy"
Nov 02 18:30:12.142008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141962    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy"
Nov 02 18:30:12.142008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141967    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy"
Nov 02 18:30:12.142008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141987    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy"
Nov 02 18:30:12.142008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.141993    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142018    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4f0549c9-4136-492d-a11a-c98e27837baf" containerName="revoke-keys"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142023    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0549c9-4136-492d-a11a-c98e27837baf" containerName="revoke-keys"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142029    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142033    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142039    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerName="cleanup"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142043    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerName="cleanup"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142144    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0549c9-4136-492d-a11a-c98e27837baf" containerName="revoke-keys"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142154    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142160    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerName="cleanup"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142168    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerName="cleanup"
Nov 02 18:30:12.142176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142177    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy"
Nov 02 18:30:12.142570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.142185    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8f6b1-4cf9-4715-8974-3cdf3a90cee6" containerName="istio-proxy"
Nov 02 18:30:12.147284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.147246    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.151116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.151080    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:30:12.152977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.152943    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"]
Nov 02 18:30:12.236968 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.236930    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wwq\" (UniqueName: \"kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.237261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.237238    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.237373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.237294    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.338856 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.338785    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.339086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.338926    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.339086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.339016    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wwq\" (UniqueName: \"kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.339487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.339452    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.341702 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.341681    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.347234 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.347206    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wwq\" (UniqueName: \"kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq\") pod \"maas-api-revoke-sub-3310df55-6fmjb\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.465773 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.465728    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:12.765193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.765165    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"]
Nov 02 18:30:12.766361 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:12.766331    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765282cd_b901_4a14_b250_20429c82b51d.slice/crio-bac29e078ab5237a3414cd3924f328e2437da52137451fdf44e5800affeeae3a WatchSource:0}: Error finding container bac29e078ab5237a3414cd3924f328e2437da52137451fdf44e5800affeeae3a: Status 404 returned error can't find the container with id bac29e078ab5237a3414cd3924f328e2437da52137451fdf44e5800affeeae3a
Nov 02 18:30:12.852059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.851996    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:12.853010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:12.852984    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:30:13.041523 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:13.041429    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb" event={"ID":"765282cd-b901-4a14-b250-20429c82b51d","Type":"ContainerStarted","Data":"bac29e078ab5237a3414cd3924f328e2437da52137451fdf44e5800affeeae3a"}
Nov 02 18:30:14.051519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:14.051484    5042 generic.go:356] "Generic (PLEG): container finished" podID="765282cd-b901-4a14-b250-20429c82b51d" containerID="b49f4aeec5a4665f244f5eebb486913e462eaa99f098087e480ea96f632370bb" exitCode=0
Nov 02 18:30:14.052005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:14.051571    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb" event={"ID":"765282cd-b901-4a14-b250-20429c82b51d","Type":"ContainerDied","Data":"b49f4aeec5a4665f244f5eebb486913e462eaa99f098087e480ea96f632370bb"}
Nov 02 18:30:14.536480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:14.536441    5042 scope.go:122] "RemoveContainer" containerID="92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00"
Nov 02 18:30:15.062955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.062879    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerStarted","Data":"3aa0fc91fabb6b7cac2868fbce0f16f39a0e8e3fff4e507ca674d1907dd54b61"}
Nov 02 18:30:15.386205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.386157    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:15.541094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.530571    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca\") pod \"765282cd-b901-4a14-b250-20429c82b51d\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") "
Nov 02 18:30:15.541438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.541401    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca\") pod \"765282cd-b901-4a14-b250-20429c82b51d\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") "
Nov 02 18:30:15.541745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.541724    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq\" (UniqueName: \"kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq\") pod \"765282cd-b901-4a14-b250-20429c82b51d\" (UID: \"765282cd-b901-4a14-b250-20429c82b51d\") "
Nov 02 18:30:15.596367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.556468    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca" pod "765282cd-b901-4a14-b250-20429c82b51d" (UID: "765282cd-b901-4a14-b250-20429c82b51d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:15.596367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.566132    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq" pod "765282cd-b901-4a14-b250-20429c82b51d" (UID: "765282cd-b901-4a14-b250-20429c82b51d"). InnerVolumeSpecName "kube-api-access-r7wwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:15.596367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.594959    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca" pod "765282cd-b901-4a14-b250-20429c82b51d" (UID: "765282cd-b901-4a14-b250-20429c82b51d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:15.596367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.595582    5042 scope.go:122] "RemoveContainer" containerID="42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3"
Nov 02 18:30:15.596912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.596519    5042 scope.go:122] "RemoveContainer" containerID="197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b"
Nov 02 18:30:15.598304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.597954    5042 scope.go:122] "RemoveContainer" containerID="a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56"
Nov 02 18:30:15.644396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.644303    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7wwq\" (UniqueName: \"kubernetes.io/projected/765282cd-b901-4a14-b250-20429c82b51d-kube-api-access-r7wwq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:15.644396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.644343    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/765282cd-b901-4a14-b250-20429c82b51d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:15.644396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:15.644360    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/765282cd-b901-4a14-b250-20429c82b51d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:16.155480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.155430    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"]
Nov 02 18:30:16.164016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.163987    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"
Nov 02 18:30:16.164192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.164055    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb" event={"ID":"765282cd-b901-4a14-b250-20429c82b51d","Type":"ContainerDied","Data":"bac29e078ab5237a3414cd3924f328e2437da52137451fdf44e5800affeeae3a"}
Nov 02 18:30:16.164192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.164122    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bac29e078ab5237a3414cd3924f328e2437da52137451fdf44e5800affeeae3a"
Nov 02 18:30:16.170561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.169602    5042 generic.go:356] "Generic (PLEG): container finished" podID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerID="3aa0fc91fabb6b7cac2868fbce0f16f39a0e8e3fff4e507ca674d1907dd54b61" exitCode=60
Nov 02 18:30:16.170561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.169735    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerDied","Data":"3aa0fc91fabb6b7cac2868fbce0f16f39a0e8e3fff4e507ca674d1907dd54b61"}
Nov 02 18:30:16.170561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.169813    5042 scope.go:122] "RemoveContainer" containerID="92a7b0f7c541b1840c60300446ab2bf7c565c7b905a530e4031dd74aca0d9b00"
Nov 02 18:30:16.591895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.567526    5042 scope.go:122] "RemoveContainer" containerID="f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f"
Nov 02 18:30:16.591895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.571095    5042 scope.go:122] "RemoveContainer" containerID="270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf"
Nov 02 18:30:16.779066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.779024    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:16.878869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.878451    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca\") pod \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") "
Nov 02 18:30:16.878869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.878656    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586\" (UniqueName: \"kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586\") pod \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\" (UID: \"13aa19a0-04cb-4da0-99dc-b4995196d1c2\") "
Nov 02 18:30:16.879559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.879536    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca" pod "13aa19a0-04cb-4da0-99dc-b4995196d1c2" (UID: "13aa19a0-04cb-4da0-99dc-b4995196d1c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:16.881365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.881337    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586" pod "13aa19a0-04cb-4da0-99dc-b4995196d1c2" (UID: "13aa19a0-04cb-4da0-99dc-b4995196d1c2"). InnerVolumeSpecName "kube-api-access-sw586". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:16.980475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.980441    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw586\" (UniqueName: \"kubernetes.io/projected/13aa19a0-04cb-4da0-99dc-b4995196d1c2-kube-api-access-sw586\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:16.980475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:16.980473    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13aa19a0-04cb-4da0-99dc-b4995196d1c2-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:17.197456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.197406    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerStarted","Data":"a7701b75ee9ae58f7583392466a03d1a283643a0c236b28a53735b40f1d95ef4"}
Nov 02 18:30:17.216123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.216088    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerStarted","Data":"182124b864918dbddb07d671e28a7e9a4b377fa1442d1ce3e726e3ba11489731"}
Nov 02 18:30:17.219726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.219700    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g" event={"ID":"13aa19a0-04cb-4da0-99dc-b4995196d1c2","Type":"ContainerDied","Data":"1752b4a2f2ea4fd549f0d4e675c66d0e2eb49e9492b0f7f8a65b49faade83ecc"}
Nov 02 18:30:17.219910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.219761    5042 scope.go:122] "RemoveContainer" containerID="3aa0fc91fabb6b7cac2868fbce0f16f39a0e8e3fff4e507ca674d1907dd54b61"
Nov 02 18:30:17.219910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.219906    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"
Nov 02 18:30:17.224522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.224476    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerStarted","Data":"14f3e695a5f1d1d9d25dca7c6c29eda23a8b1ee486369dd7486d1ffeaf3fd342"}
Nov 02 18:30:17.298590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.298558    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"]
Nov 02 18:30:17.309412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.309365    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b97c6-29894070-lts4g"]
Nov 02 18:30:17.556372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.556299    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" path="/var/lib/kubelet/pods/13aa19a0-04cb-4da0-99dc-b4995196d1c2/volumes"
Nov 02 18:30:17.633735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.633690    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:30:17.679653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.679610    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:30:17.814491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.814386    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume\") pod \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") "
Nov 02 18:30:17.814491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.814428    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867\" (UniqueName: \"kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867\") pod \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") "
Nov 02 18:30:17.814491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.814471    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g\" (UniqueName: \"kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g\") pod \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\" (UID: \"8154703c-cc6b-47c3-b46f-c38c3ef721cb\") "
Nov 02 18:30:17.814821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.814613    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume\") pod \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\" (UID: \"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23\") "
Nov 02 18:30:17.815025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.815000    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume" pod "8154703c-cc6b-47c3-b46f-c38c3ef721cb" (UID: "8154703c-cc6b-47c3-b46f-c38c3ef721cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:17.815201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.815027    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume" pod "0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" (UID: "0c2638b1-ae0f-4bec-996b-1ccd5f40bb23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:17.817536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.817512    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867" pod "0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" (UID: "0c2638b1-ae0f-4bec-996b-1ccd5f40bb23"). InnerVolumeSpecName "kube-api-access-rg867". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:17.818283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.818257    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g" pod "8154703c-cc6b-47c3-b46f-c38c3ef721cb" (UID: "8154703c-cc6b-47c3-b46f-c38c3ef721cb"). InnerVolumeSpecName "kube-api-access-l7c5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:17.917970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.917090    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8154703c-cc6b-47c3-b46f-c38c3ef721cb-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:17.917970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.917125    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg867\" (UniqueName: \"kubernetes.io/projected/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-kube-api-access-rg867\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:17.917970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.917140    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7c5g\" (UniqueName: \"kubernetes.io/projected/8154703c-cc6b-47c3-b46f-c38c3ef721cb-kube-api-access-l7c5g\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:17.917970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:17.917153    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.245213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.245166    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"]
Nov 02 18:30:18.255399 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.255359    5042 generic.go:356] "Generic (PLEG): container finished" podID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerID="f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085" exitCode=0
Nov 02 18:30:18.256283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.256248    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" event={"ID":"8154703c-cc6b-47c3-b46f-c38c3ef721cb","Type":"ContainerDied","Data":"f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085"}
Nov 02 18:30:18.256425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.256321    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w" event={"ID":"8154703c-cc6b-47c3-b46f-c38c3ef721cb","Type":"ContainerDied","Data":"c9b7d40a5b7a21a0d880150abaf1d682e334d5889b3b66a69dcb08a613aa9aed"}
Nov 02 18:30:18.256425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.256354    5042 scope.go:122] "RemoveContainer" containerID="f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085"
Nov 02 18:30:18.256606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.256471    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"
Nov 02 18:30:18.263920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.263204    5042 generic.go:356] "Generic (PLEG): container finished" podID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerID="182124b864918dbddb07d671e28a7e9a4b377fa1442d1ce3e726e3ba11489731" exitCode=60
Nov 02 18:30:18.264568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.264262    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerDied","Data":"182124b864918dbddb07d671e28a7e9a4b377fa1442d1ce3e726e3ba11489731"}
Nov 02 18:30:18.265427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.265395    5042 scope.go:122] "RemoveContainer" containerID="182124b864918dbddb07d671e28a7e9a4b377fa1442d1ce3e726e3ba11489731"
Nov 02 18:30:18.265781 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:18.265760    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k_odh-ai-gateway-infra(f15c7d0e-0668-454b-8c09-9ffa2745c28c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c"
Nov 02 18:30:18.272866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.269691    5042 generic.go:356] "Generic (PLEG): container finished" podID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerID="d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8" exitCode=0
Nov 02 18:30:18.272866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.269864    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" event={"ID":"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23","Type":"ContainerDied","Data":"d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8"}
Nov 02 18:30:18.272866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.269910    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v" event={"ID":"0c2638b1-ae0f-4bec-996b-1ccd5f40bb23","Type":"ContainerDied","Data":"c11d859b6ce1d2a382e0f48e7b0be5d05c793bdbc6f64baf1d9a86d0afc245d6"}
Nov 02 18:30:18.272866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.269986    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"
Nov 02 18:30:18.272866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.270324    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"]
Nov 02 18:30:18.293896 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.293856    5042 scope.go:122] "RemoveContainer" containerID="f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085"
Nov 02 18:30:18.295658 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:18.295511    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085\": container with ID starting with f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085 not found: ID does not exist" containerID="f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085"
Nov 02 18:30:18.295658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.295558    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085"} err="failed to get container status \"f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085\": rpc error: code = NotFound desc = could not find container \"f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085\": container with ID starting with f281a46a61c87d20e78c7d712958756cc3f73e8807b2954a191bf68fbd904085 not found: ID does not exist"
Nov 02 18:30:18.295658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.295586    5042 scope.go:122] "RemoveContainer" containerID="a15f8af2ac1ea97ce8cc2989763722b0b74ae72b2f66073f268e3401d7824d56"
Nov 02 18:30:18.301212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.301182    5042 generic.go:356] "Generic (PLEG): container finished" podID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerID="14f3e695a5f1d1d9d25dca7c6c29eda23a8b1ee486369dd7486d1ffeaf3fd342" exitCode=60
Nov 02 18:30:18.301401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.301303    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerDied","Data":"14f3e695a5f1d1d9d25dca7c6c29eda23a8b1ee486369dd7486d1ffeaf3fd342"}
Nov 02 18:30:18.321680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.316507    5042 generic.go:356] "Generic (PLEG): container finished" podID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerID="fa685f360fbaed7b9f7f80c7fa35d1c5e0d5d448b7269c0ebcfba019510ddc6f" exitCode=60
Nov 02 18:30:18.321680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.316654    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" event={"ID":"cba10682-eff3-45f2-91bf-33bb1df21c2e","Type":"ContainerDied","Data":"fa685f360fbaed7b9f7f80c7fa35d1c5e0d5d448b7269c0ebcfba019510ddc6f"}
Nov 02 18:30:18.370990 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.370943    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"]
Nov 02 18:30:18.418481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.416441    5042 scope.go:122] "RemoveContainer" containerID="d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8"
Nov 02 18:30:18.432877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.428875    5042 scope.go:122] "RemoveContainer" containerID="fa685f360fbaed7b9f7f80c7fa35d1c5e0d5d448b7269c0ebcfba019510ddc6f"
Nov 02 18:30:18.432877 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:18.430353    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx_odh-ai-gateway-infra(cba10682-eff3-45f2-91bf-33bb1df21c2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e"
Nov 02 18:30:18.445012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.444682    5042 generic.go:356] "Generic (PLEG): container finished" podID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerID="22ce705268baf6296db53ae446ba53d5ed089e9dc68db78795bede93eae5e9dd" exitCode=60
Nov 02 18:30:18.445012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.444873    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerDied","Data":"22ce705268baf6296db53ae446ba53d5ed089e9dc68db78795bede93eae5e9dd"}
Nov 02 18:30:18.445516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.445489    5042 scope.go:122] "RemoveContainer" containerID="22ce705268baf6296db53ae446ba53d5ed089e9dc68db78795bede93eae5e9dd"
Nov 02 18:30:18.446389 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:18.446360    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr_odh-ai-gateway-infra(b26ebda4-2401-4e89-9668-1f0f683b292a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a"
Nov 02 18:30:18.462882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.450289    5042 scope.go:122] "RemoveContainer" containerID="d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8"
Nov 02 18:30:18.463898 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:18.463860    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8\": container with ID starting with d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8 not found: ID does not exist" containerID="d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8"
Nov 02 18:30:18.464188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.464138    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8"} err="failed to get container status \"d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8\": rpc error: code = NotFound desc = could not find container \"d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8\": container with ID starting with d9d63d5fa246e65fe8c052cb0d1e7506dc25d05d9e309f5af0dad4738711caa8 not found: ID does not exist"
Nov 02 18:30:18.464400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.464382    5042 scope.go:122] "RemoveContainer" containerID="197205b5848acf8ca9e3d445e70103bab7d017c4bf263bd58015196b41db2b8b"
Nov 02 18:30:18.464682 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.464667    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"]
Nov 02 18:30:18.473322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.473049    5042 generic.go:356] "Generic (PLEG): container finished" podID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerID="a7701b75ee9ae58f7583392466a03d1a283643a0c236b28a53735b40f1d95ef4" exitCode=60
Nov 02 18:30:18.473322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.473115    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerDied","Data":"a7701b75ee9ae58f7583392466a03d1a283643a0c236b28a53735b40f1d95ef4"}
Nov 02 18:30:18.479195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.478888    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c3c1f-df9774475-2kz8w"]
Nov 02 18:30:18.505623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.505550    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"]
Nov 02 18:30:18.509501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.509471    5042 scope.go:122] "RemoveContainer" containerID="270e900931a133d3540ef583bf467645328a7f89970e8cca9856ea9e9885cfdf"
Nov 02 18:30:18.510556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.510526    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c3c1f-7d6c46555-dww8v"]
Nov 02 18:30:18.551516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.551487    5042 scope.go:122] "RemoveContainer" containerID="f059af6ed0d4f7f744e2ec42ead686f34271cb495e857bae237ca0212b95045f"
Nov 02 18:30:18.586273 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:18.586205    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:30:18.654133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.632149    5042 scope.go:122] "RemoveContainer" containerID="42deb61fe1dba6209e25852a86e610aef63d8a1f8af62840a225336d95a90ff3"
Nov 02 18:30:18.654133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.635381    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:18.722129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.722087    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd\" (UniqueName: \"kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd\") pod \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") "
Nov 02 18:30:18.723993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.723964    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca\") pod \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\" (UID: \"dbd96234-118e-41f0-aa6b-6c778e49fcc4\") "
Nov 02 18:30:18.731948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.727047    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca" pod "dbd96234-118e-41f0-aa6b-6c778e49fcc4" (UID: "dbd96234-118e-41f0-aa6b-6c778e49fcc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:18.732152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.730825    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd" pod "dbd96234-118e-41f0-aa6b-6c778e49fcc4" (UID: "dbd96234-118e-41f0-aa6b-6c778e49fcc4"). InnerVolumeSpecName "kube-api-access-57bmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:18.827934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.817676    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:18.832723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.832687    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57bmd\" (UniqueName: \"kubernetes.io/projected/dbd96234-118e-41f0-aa6b-6c778e49fcc4-kube-api-access-57bmd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.832723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.832721    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbd96234-118e-41f0-aa6b-6c778e49fcc4-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:18.936539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.936487    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca\") pod \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") "
Nov 02 18:30:18.936762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.936602    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb\" (UniqueName: \"kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb\") pod \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\" (UID: \"0dd837fe-25e5-4fbe-b677-3eb53110fa5e\") "
Nov 02 18:30:18.940298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.940263    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca" pod "0dd837fe-25e5-4fbe-b677-3eb53110fa5e" (UID: "0dd837fe-25e5-4fbe-b677-3eb53110fa5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:18.941028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:18.940990    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb" pod "0dd837fe-25e5-4fbe-b677-3eb53110fa5e" (UID: "0dd837fe-25e5-4fbe-b677-3eb53110fa5e"). InnerVolumeSpecName "kube-api-access-6sqzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:19.046933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.046627    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.046933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.046673    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sqzb\" (UniqueName: \"kubernetes.io/projected/0dd837fe-25e5-4fbe-b677-3eb53110fa5e-kube-api-access-6sqzb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:19.111748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.111706    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"]
Nov 02 18:30:19.112263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.112226    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy" containerID="cri-o://07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea" gracePeriod=30
Nov 02 18:30:19.437551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.437445    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" containerID="cri-o://a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575" gracePeriod=45
Nov 02 18:30:19.488493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.488390    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk" event={"ID":"dbd96234-118e-41f0-aa6b-6c778e49fcc4","Type":"ContainerDied","Data":"2b57b6eaf0baa481c5e9a81df3cbf10c4b4540ce0f2a4cbd889dff3a01f680c0"}
Nov 02 18:30:19.488493 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.488431    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"
Nov 02 18:30:19.488808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.488464    5042 scope.go:122] "RemoveContainer" containerID="14f3e695a5f1d1d9d25dca7c6c29eda23a8b1ee486369dd7486d1ffeaf3fd342"
Nov 02 18:30:19.496558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.495852    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z" event={"ID":"0dd837fe-25e5-4fbe-b677-3eb53110fa5e","Type":"ContainerDied","Data":"4b6115ad095f971c7793a0f64aadfb567fa187b508bbdef36d768802a6e36bf3"}
Nov 02 18:30:19.496558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.495981    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"
Nov 02 18:30:19.555102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.555022    5042 scope.go:122] "RemoveContainer" containerID="a7701b75ee9ae58f7583392466a03d1a283643a0c236b28a53735b40f1d95ef4"
Nov 02 18:30:19.555372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.555343    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" path="/var/lib/kubelet/pods/0c2638b1-ae0f-4bec-996b-1ccd5f40bb23/volumes"
Nov 02 18:30:19.556084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.556063    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" path="/var/lib/kubelet/pods/8154703c-cc6b-47c3-b46f-c38c3ef721cb/volumes"
Nov 02 18:30:19.556743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.556677    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"]
Nov 02 18:30:19.556899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.556884    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"]
Nov 02 18:30:19.556970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.556914    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-9165c3-29894070-ddgzk"]
Nov 02 18:30:19.575654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.575592    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"]
Nov 02 18:30:19.594402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.586719    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"]
Nov 02 18:30:19.594402 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.591967    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-1e2281-29894070-xsd8z"]
Nov 02 18:30:19.811865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.811220    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:19.910539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.910238    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:19.912395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.912363    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:19.970810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.970736    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:30:19.991952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.991915    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7\" (UniqueName: \"kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7\") pod \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") "
Nov 02 18:30:19.992226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.992201    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca\") pod \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\" (UID: \"f15c7d0e-0668-454b-8c09-9ffa2745c28c\") "
Nov 02 18:30:19.993541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:19.993509    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca" pod "f15c7d0e-0668-454b-8c09-9ffa2745c28c" (UID: "f15c7d0e-0668-454b-8c09-9ffa2745c28c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:20.002997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.002953    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7" pod "f15c7d0e-0668-454b-8c09-9ffa2745c28c" (UID: "f15c7d0e-0668-454b-8c09-9ffa2745c28c"). InnerVolumeSpecName "kube-api-access-txzr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:20.093734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.093697    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca\") pod \"b26ebda4-2401-4e89-9668-1f0f683b292a\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") "
Nov 02 18:30:20.093960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.093750    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4\" (UniqueName: \"kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4\") pod \"cba10682-eff3-45f2-91bf-33bb1df21c2e\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") "
Nov 02 18:30:20.093960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.093788    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49\" (UniqueName: \"kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49\") pod \"b26ebda4-2401-4e89-9668-1f0f683b292a\" (UID: \"b26ebda4-2401-4e89-9668-1f0f683b292a\") "
Nov 02 18:30:20.094197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094158    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca" pod "b26ebda4-2401-4e89-9668-1f0f683b292a" (UID: "b26ebda4-2401-4e89-9668-1f0f683b292a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:20.094197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094178    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca\") pod \"cba10682-eff3-45f2-91bf-33bb1df21c2e\" (UID: \"cba10682-eff3-45f2-91bf-33bb1df21c2e\") "
Nov 02 18:30:20.094453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094240    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache\") pod \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") "
Nov 02 18:30:20.094453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094380    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir\") pod \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") "
Nov 02 18:30:20.094453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094420    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm\") pod \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") "
Nov 02 18:30:20.094636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094468    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home\") pod \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") "
Nov 02 18:30:20.094636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094521    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw\" (UniqueName: \"kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw\") pod \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") "
Nov 02 18:30:20.094636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094555    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs\") pod \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\" (UID: \"6a4c8897-3567-48cc-84fd-1f6f6ab64245\") "
Nov 02 18:30:20.094636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094579    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca" pod "cba10682-eff3-45f2-91bf-33bb1df21c2e" (UID: "cba10682-eff3-45f2-91bf-33bb1df21c2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:20.094892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.094814    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home" pod "6a4c8897-3567-48cc-84fd-1f6f6ab64245" (UID: "6a4c8897-3567-48cc-84fd-1f6f6ab64245"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:20.095071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.095052    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir" pod "6a4c8897-3567-48cc-84fd-1f6f6ab64245" (UID: "6a4c8897-3567-48cc-84fd-1f6f6ab64245"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:20.095255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.095237    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f15c7d0e-0668-454b-8c09-9ffa2745c28c-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.095401 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.095386    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzr7\" (UniqueName: \"kubernetes.io/projected/f15c7d0e-0668-454b-8c09-9ffa2745c28c-kube-api-access-txzr7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.095513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.095498    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b26ebda4-2401-4e89-9668-1f0f683b292a-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.095664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.095648    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba10682-eff3-45f2-91bf-33bb1df21c2e-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.096764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.096745    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.096909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.095391    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache" pod "6a4c8897-3567-48cc-84fd-1f6f6ab64245" (UID: "6a4c8897-3567-48cc-84fd-1f6f6ab64245"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:20.097264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.097210    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49" pod "b26ebda4-2401-4e89-9668-1f0f683b292a" (UID: "b26ebda4-2401-4e89-9668-1f0f683b292a"). InnerVolumeSpecName "kube-api-access-c7h49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:20.097936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.097902    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm" pod "6a4c8897-3567-48cc-84fd-1f6f6ab64245" (UID: "6a4c8897-3567-48cc-84fd-1f6f6ab64245"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:20.098292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.098273    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4" pod "cba10682-eff3-45f2-91bf-33bb1df21c2e" (UID: "cba10682-eff3-45f2-91bf-33bb1df21c2e"). InnerVolumeSpecName "kube-api-access-ngng4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:20.099283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.099257    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw" pod "6a4c8897-3567-48cc-84fd-1f6f6ab64245" (UID: "6a4c8897-3567-48cc-84fd-1f6f6ab64245"). InnerVolumeSpecName "kube-api-access-mk8sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:20.099450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.099429    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs" pod "6a4c8897-3567-48cc-84fd-1f6f6ab64245" (UID: "6a4c8897-3567-48cc-84fd-1f6f6ab64245"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.227938    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngng4\" (UniqueName: \"kubernetes.io/projected/cba10682-eff3-45f2-91bf-33bb1df21c2e-kube-api-access-ngng4\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.227974    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7h49\" (UniqueName: \"kubernetes.io/projected/b26ebda4-2401-4e89-9668-1f0f683b292a-kube-api-access-c7h49\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.227993    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.228006    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.228022    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a4c8897-3567-48cc-84fd-1f6f6ab64245-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.228035    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8sw\" (UniqueName: \"kubernetes.io/projected/6a4c8897-3567-48cc-84fd-1f6f6ab64245-kube-api-access-mk8sw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.228158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.228049    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a4c8897-3567-48cc-84fd-1f6f6ab64245-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:20.520945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.520023    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"
Nov 02 18:30:20.520945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.520058    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx" event={"ID":"cba10682-eff3-45f2-91bf-33bb1df21c2e","Type":"ContainerDied","Data":"bd58e4c915e39d6837e76b49bc5de47fc86510772d3fe96acbd3564198bb6ec1"}
Nov 02 18:30:20.520945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.520141    5042 scope.go:122] "RemoveContainer" containerID="fa685f360fbaed7b9f7f80c7fa35d1c5e0d5d448b7269c0ebcfba019510ddc6f"
Nov 02 18:30:20.522575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.522538    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"
Nov 02 18:30:20.522903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.522542    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr" event={"ID":"b26ebda4-2401-4e89-9668-1f0f683b292a","Type":"ContainerDied","Data":"94c038e569c914e5df1cb3bee3a5fe187d603b42f0db987d2954ee62ed19f8a3"}
Nov 02 18:30:20.526545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.526519    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"
Nov 02 18:30:20.526663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.526519    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k" event={"ID":"f15c7d0e-0668-454b-8c09-9ffa2745c28c","Type":"ContainerDied","Data":"92f74336a1cf15f4f4f976902f9a39cbee5023dc9f97df9aec85534104592e7b"}
Nov 02 18:30:20.528439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.528411    5042 generic.go:356] "Generic (PLEG): container finished" podID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerID="a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575" exitCode=0
Nov 02 18:30:20.528550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.528448    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"
Nov 02 18:30:20.528550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.528538    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" event={"ID":"6a4c8897-3567-48cc-84fd-1f6f6ab64245","Type":"ContainerDied","Data":"a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575"}
Nov 02 18:30:20.528689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.528568    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" event={"ID":"6a4c8897-3567-48cc-84fd-1f6f6ab64245","Type":"ContainerDied","Data":"b5aa627a45ac5966d16bc8e745f83f54d38b9d538f0ad66a3cd4c8088dca82e0"}
Nov 02 18:30:20.536445 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.535539    5042 scope.go:122] "RemoveContainer" containerID="22ce705268baf6296db53ae446ba53d5ed089e9dc68db78795bede93eae5e9dd"
Nov 02 18:30:20.548040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.548019    5042 scope.go:122] "RemoveContainer" containerID="182124b864918dbddb07d671e28a7e9a4b377fa1442d1ce3e726e3ba11489731"
Nov 02 18:30:20.558594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.558573    5042 scope.go:122] "RemoveContainer" containerID="a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575"
Nov 02 18:30:20.565539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.565453    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"]
Nov 02 18:30:20.570489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.570456    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-7771bf-29894070-w7bvx"]
Nov 02 18:30:20.572261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.572236    5042 scope.go:122] "RemoveContainer" containerID="a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575"
Nov 02 18:30:20.572648 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:20.572622    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575\": container with ID starting with a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575 not found: ID does not exist" containerID="a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575"
Nov 02 18:30:20.572765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.572657    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575"} err="failed to get container status \"a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575\": rpc error: code = NotFound desc = could not find container \"a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575\": container with ID starting with a75a425513f6a85249ed82e32c0b7d323f47a016b69f1ab62adfce2df9b82575 not found: ID does not exist"
Nov 02 18:30:20.579280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.579248    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"]
Nov 02 18:30:20.595754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.595707    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-66eed1-29894070-rs7gr"]
Nov 02 18:30:20.606277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.606236    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"]
Nov 02 18:30:20.613561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.613517    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-1ef96f-29894070-hhg7k"]
Nov 02 18:30:20.619643 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.619606    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"]
Nov 02 18:30:20.623202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.623171    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p"]
Nov 02 18:30:20.781769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:20.781646    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-a37cf0/simulator-w3-a37cf0-kserve-69c44c9f58-7rt6p" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:30:21.545061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:21.545027    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" path="/var/lib/kubelet/pods/0dd837fe-25e5-4fbe-b677-3eb53110fa5e/volumes"
Nov 02 18:30:21.545564 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:21.545549    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" path="/var/lib/kubelet/pods/6a4c8897-3567-48cc-84fd-1f6f6ab64245/volumes"
Nov 02 18:30:21.545984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:21.545969    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" path="/var/lib/kubelet/pods/b26ebda4-2401-4e89-9668-1f0f683b292a/volumes"
Nov 02 18:30:21.546672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:21.546656    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" path="/var/lib/kubelet/pods/cba10682-eff3-45f2-91bf-33bb1df21c2e/volumes"
Nov 02 18:30:21.547080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:21.547062    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" path="/var/lib/kubelet/pods/dbd96234-118e-41f0-aa6b-6c778e49fcc4/volumes"
Nov 02 18:30:21.547422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:21.547410    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" path="/var/lib/kubelet/pods/f15c7d0e-0668-454b-8c09-9ffa2745c28c/volumes"
Nov 02 18:30:24.466359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.466330    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 18:30:24.487522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487477    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.487709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487554    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.487709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487662    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.487882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487722    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.487882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487800    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.488000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487911    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.488000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.487953    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.488115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.488005    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.488115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.488078    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert\") pod \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\" (UID: \"9cf9580b-aed3-4788-a8f9-fb1427610c9a\") "
Nov 02 18:30:24.489480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.489451    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:24.490034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.489991    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:24.490179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.490137    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:24.490179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.490150    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:24.490639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.490427    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:24.492810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.492780    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "kube-api-access-mbjcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:24.494931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.494883    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:30:24.495167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.495139    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:24.497790 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.497765    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token" pod "9cf9580b-aed3-4788-a8f9-fb1427610c9a" (UID: "9cf9580b-aed3-4788-a8f9-fb1427610c9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:24.567449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.567411    5042 generic.go:356] "Generic (PLEG): container finished" podID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerID="07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea" exitCode=0
Nov 02 18:30:24.567638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.567464    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"
Nov 02 18:30:24.567638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.567500    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" event={"ID":"9cf9580b-aed3-4788-a8f9-fb1427610c9a","Type":"ContainerDied","Data":"07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea"}
Nov 02 18:30:24.567638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.567569    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7" event={"ID":"9cf9580b-aed3-4788-a8f9-fb1427610c9a","Type":"ContainerDied","Data":"91760b23738c91107a2cca3a6a11dd2dc2e5bc67961ad9dc9a3c9f3f32b53950"}
Nov 02 18:30:24.567638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.567598    5042 scope.go:122] "RemoveContainer" containerID="07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea"
Nov 02 18:30:24.580852 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.580807    5042 scope.go:122] "RemoveContainer" containerID="546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"
Nov 02 18:30:24.589483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589454    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589481    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589494    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589504    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589515    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjcj\" (UniqueName: \"kubernetes.io/projected/9cf9580b-aed3-4788-a8f9-fb1427610c9a-kube-api-access-mbjcj\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589524    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589533    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589542    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cf9580b-aed3-4788-a8f9-fb1427610c9a-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.589666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.589550    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cf9580b-aed3-4788-a8f9-fb1427610c9a-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:24.596055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.596028    5042 scope.go:122] "RemoveContainer" containerID="07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea"
Nov 02 18:30:24.596488 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:24.596457    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea\": container with ID starting with 07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea not found: ID does not exist" containerID="07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea"
Nov 02 18:30:24.596595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.596503    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea"} err="failed to get container status \"07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea\": rpc error: code = NotFound desc = could not find container \"07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea\": container with ID starting with 07651cc8cdbea277f2a56cd980574524dd87258d2a701c5a49e1cbe144dda0ea not found: ID does not exist"
Nov 02 18:30:24.596595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.596529    5042 scope.go:122] "RemoveContainer" containerID="546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"
Nov 02 18:30:24.597182 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:24.597152    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8\": container with ID starting with 546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8 not found: ID does not exist" containerID="546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"
Nov 02 18:30:24.597328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.597192    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8"} err="failed to get container status \"546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8\": rpc error: code = NotFound desc = could not find container \"546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8\": container with ID starting with 546b1ce3bf756928e9718d38a61748b98f770ae6ed3d96c122508da1e0f4a3c8 not found: ID does not exist"
Nov 02 18:30:24.597483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.597466    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"]
Nov 02 18:30:24.602934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:24.602904    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-86f5f0-openshift-default-6c8789d857-9mzg7"]
Nov 02 18:30:25.544303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:25.544270    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" path="/var/lib/kubelet/pods/9cf9580b-aed3-4788-a8f9-fb1427610c9a/volumes"
Nov 02 18:30:32.929595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:32.929555    5042 scope.go:122] "RemoveContainer" containerID="5a93f28c488532c07a1309ad021cd8157d6d3c85380f5cb1f46a78e64109665d"
Nov 02 18:30:32.941708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:32.941679    5042 scope.go:122] "RemoveContainer" containerID="4102bf49ede5e568329243e2f141cffa5e85a64114b0a1a8c016a0a9b26e7658"
Nov 02 18:30:34.371484 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.371451    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"]
Nov 02 18:30:34.372039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372024    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy"
Nov 02 18:30:34.372095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372040    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy"
Nov 02 18:30:34.372095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372052    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:34.372095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372058    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:34.372095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372073    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:34.372095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372078    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:34.372095 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372093    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372098    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372107    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372112    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372118    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372123    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372132    5042 cpu_manager.go:414] "containerMap: removing container" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372136    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372144    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372148    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372154    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372158    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372165    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372169    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372178    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372182    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372189    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372193    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372200    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372204    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99b18a5-b3a5-40a4-b968-68499c02e2d1" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372214    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372219    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372225    5042 cpu_manager.go:414] "containerMap: removing container" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372230    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372238    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372243    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372250    5042 cpu_manager.go:414] "containerMap: removing container" podUID="765282cd-b901-4a14-b250-20429c82b51d" containerName="revoke-keys"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372255    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765282cd-b901-4a14-b250-20429c82b51d" containerName="revoke-keys"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372260    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerName="payload-processing"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372265    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerName="payload-processing"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372270    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372275    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372281    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372284    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372295    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.372287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372299    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372383    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372390    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372396    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372405    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4c8897-3567-48cc-84fd-1f6f6ab64245" containerName="main"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372411    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372416    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372421    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372426    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372431    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372435    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372441    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf9580b-aed3-4788-a8f9-fb1427610c9a" containerName="istio-proxy"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372447    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd837fe-25e5-4fbe-b677-3eb53110fa5e" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372453    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04ba0c7-e939-4825-81be-080c7b4139d9" containerName="istio-proxy"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372459    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372465    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8154703c-cc6b-47c3-b46f-c38c3ef721cb" containerName="payload-pre-processing"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372471    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="765282cd-b901-4a14-b250-20429c82b51d" containerName="revoke-keys"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372476    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2638b1-ae0f-4bec-996b-1ccd5f40bb23" containerName="payload-processing"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372480    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372485    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:34.373422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.372491    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:34.380069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.380036    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.382592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.382556    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"]
Nov 02 18:30:34.446116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.446079    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.446305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.446239    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.446355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.446334    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92tt\" (UniqueName: \"kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.460281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.460245    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:30:34.460460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.460301    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:30:34.547500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.547462    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.547697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.547583    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.547697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.547658    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92tt\" (UniqueName: \"kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.548388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.548369    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.550499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.550475    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.556117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.556089    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92tt\" (UniqueName: \"kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt\") pod \"maas-api-revoke-sub-f4c3245f-zmhch\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:34.701172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:34.701132    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:35.026361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:35.026327    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"]
Nov 02 18:30:35.027628 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:35.027587    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75f8abb_f63e_4d4f_947f_399cf2748150.slice/crio-bced6978735527b9d3efd20f50fc1f6828ebc6c18c00155e2a6683ee866953f5 WatchSource:0}: Error finding container bced6978735527b9d3efd20f50fc1f6828ebc6c18c00155e2a6683ee866953f5: Status 404 returned error can't find the container with id bced6978735527b9d3efd20f50fc1f6828ebc6c18c00155e2a6683ee866953f5
Nov 02 18:30:35.756752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:35.756708    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch" event={"ID":"e75f8abb-f63e-4d4f-947f-399cf2748150","Type":"ContainerStarted","Data":"234996f67d4dc48f400325fa227ff43b188fb0e7bcc98a270b74b77526e8cd28"}
Nov 02 18:30:35.757240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:35.756775    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch" event={"ID":"e75f8abb-f63e-4d4f-947f-399cf2748150","Type":"ContainerStarted","Data":"bced6978735527b9d3efd20f50fc1f6828ebc6c18c00155e2a6683ee866953f5"}
Nov 02 18:30:35.776339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:35.776276    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch" podStartSLOduration=1.776255973 podStartE2EDuration="1.776255973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:35.772791673 +0000 UTC m=+3448.352201060" watchObservedRunningTime="2026-11-02 18:30:35.776255973 +0000 UTC m=+3448.355665360"
Nov 02 18:30:36.767557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:36.767512    5042 generic.go:356] "Generic (PLEG): container finished" podID="e75f8abb-f63e-4d4f-947f-399cf2748150" containerID="234996f67d4dc48f400325fa227ff43b188fb0e7bcc98a270b74b77526e8cd28" exitCode=0
Nov 02 18:30:36.768037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:36.767579    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch" event={"ID":"e75f8abb-f63e-4d4f-947f-399cf2748150","Type":"ContainerDied","Data":"234996f67d4dc48f400325fa227ff43b188fb0e7bcc98a270b74b77526e8cd28"}
Nov 02 18:30:37.953005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:37.952979    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:38.094556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.094454    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca\") pod \"e75f8abb-f63e-4d4f-947f-399cf2748150\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") "
Nov 02 18:30:38.094750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.094661    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt\" (UniqueName: \"kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt\") pod \"e75f8abb-f63e-4d4f-947f-399cf2748150\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") "
Nov 02 18:30:38.094750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.094727    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca\") pod \"e75f8abb-f63e-4d4f-947f-399cf2748150\" (UID: \"e75f8abb-f63e-4d4f-947f-399cf2748150\") "
Nov 02 18:30:38.094907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.094878    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca" pod "e75f8abb-f63e-4d4f-947f-399cf2748150" (UID: "e75f8abb-f63e-4d4f-947f-399cf2748150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:38.095476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.095415    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:38.097163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.097137    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca" pod "e75f8abb-f63e-4d4f-947f-399cf2748150" (UID: "e75f8abb-f63e-4d4f-947f-399cf2748150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:38.097447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.097420    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt" pod "e75f8abb-f63e-4d4f-947f-399cf2748150" (UID: "e75f8abb-f63e-4d4f-947f-399cf2748150"). InnerVolumeSpecName "kube-api-access-m92tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:38.203676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.198610    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e75f8abb-f63e-4d4f-947f-399cf2748150-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:38.203676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.198663    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m92tt\" (UniqueName: \"kubernetes.io/projected/e75f8abb-f63e-4d4f-947f-399cf2748150-kube-api-access-m92tt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:38.787122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.787092    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"
Nov 02 18:30:38.787362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.787122    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch" event={"ID":"e75f8abb-f63e-4d4f-947f-399cf2748150","Type":"ContainerDied","Data":"bced6978735527b9d3efd20f50fc1f6828ebc6c18c00155e2a6683ee866953f5"}
Nov 02 18:30:38.787362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.787185    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bced6978735527b9d3efd20f50fc1f6828ebc6c18c00155e2a6683ee866953f5"
Nov 02 18:30:38.978915 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:38.978865    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"]
Nov 02 18:30:39.006621 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:39.006575    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"]
Nov 02 18:30:46.330367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.330321    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"]
Nov 02 18:30:46.331100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331080    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331102    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331130    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331136    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331155    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331162    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331172    5042 cpu_manager.go:414] "containerMap: removing container" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:46.331175 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331177    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331189    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e75f8abb-f63e-4d4f-947f-399cf2748150" containerName="revoke-keys"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331194    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e75f8abb-f63e-4d4f-947f-399cf2748150" containerName="revoke-keys"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331284    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26ebda4-2401-4e89-9668-1f0f683b292a" containerName="cleanup"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331295    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e75f8abb-f63e-4d4f-947f-399cf2748150" containerName="revoke-keys"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331302    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="13aa19a0-04cb-4da0-99dc-b4995196d1c2" containerName="cleanup"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331309    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f15c7d0e-0668-454b-8c09-9ffa2745c28c" containerName="cleanup"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331315    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd96234-118e-41f0-aa6b-6c778e49fcc4" containerName="cleanup"
Nov 02 18:30:46.331404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.331323    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cba10682-eff3-45f2-91bf-33bb1df21c2e" containerName="cleanup"
Nov 02 18:30:46.335511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.335479    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.340739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.340710    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:30:46.341884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.341819    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:30:46.343678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.343646    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"]
Nov 02 18:30:46.487595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.487547    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.487778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.487689    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.487936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.487789    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fmpz\" (UniqueName: \"kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.589882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.589754    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.590069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.589879    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.590069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.589941    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fmpz\" (UniqueName: \"kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.590551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.590524    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.592759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.592726    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.599514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.599479    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fmpz\" (UniqueName: \"kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz\") pod \"maas-api-revoke-sub-70121e82-2kmv6\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.654528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.654486    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:46.953042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:46.953003    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"]
Nov 02 18:30:46.953190 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:46.953005    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12bbf0b_c06f_4d94_831b_1b4d4a3a1c83.slice/crio-cd84a729f6b31fbc0ff729ff673413bc6b9383faf8a9aa456a90fb86e68cc739 WatchSource:0}: Error finding container cd84a729f6b31fbc0ff729ff673413bc6b9383faf8a9aa456a90fb86e68cc739: Status 404 returned error can't find the container with id cd84a729f6b31fbc0ff729ff673413bc6b9383faf8a9aa456a90fb86e68cc739
Nov 02 18:30:47.895657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:47.895609    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6" event={"ID":"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83","Type":"ContainerStarted","Data":"37c1d538f752a1e7b4d4e08cdf204108f1682de656537c46e0e7deaf85620d76"}
Nov 02 18:30:47.896118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:47.895683    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6" event={"ID":"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83","Type":"ContainerStarted","Data":"cd84a729f6b31fbc0ff729ff673413bc6b9383faf8a9aa456a90fb86e68cc739"}
Nov 02 18:30:47.917461 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:47.917408    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6" podStartSLOduration=1.917392282 podStartE2EDuration="1.917392282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:47.916676761 +0000 UTC m=+3460.496086148" watchObservedRunningTime="2026-11-02 18:30:47.917392282 +0000 UTC m=+3460.496801668"
Nov 02 18:30:48.567425 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:48.567387    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:30:48.743561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.743522    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"]
Nov 02 18:30:48.748000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.747971    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.754721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.754685    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"]
Nov 02 18:30:48.817899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.817764    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.818092 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.817956    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgj5\" (UniqueName: \"kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.818170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.818149    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.905545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.905503    5042 generic.go:356] "Generic (PLEG): container finished" podID="c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" containerID="37c1d538f752a1e7b4d4e08cdf204108f1682de656537c46e0e7deaf85620d76" exitCode=0
Nov 02 18:30:48.906014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.905566    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6" event={"ID":"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83","Type":"ContainerDied","Data":"37c1d538f752a1e7b4d4e08cdf204108f1682de656537c46e0e7deaf85620d76"}
Nov 02 18:30:48.919203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.919164    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.919396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.919296    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.919396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.919388    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgj5\" (UniqueName: \"kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.920116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.920032    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.922499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.922465    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:48.929088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:48.929053    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgj5\" (UniqueName: \"kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5\") pod \"maas-api-revoke-sub-f3ebc8f8-g6dtd\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:49.068107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:49.068012    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:49.351023 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:49.350990    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"]
Nov 02 18:30:49.352226 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:49.352182    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604905ae_308b_4d5f_8def_7c9ecbf28d41.slice/crio-c19dc2dc0751df0cf8402b75e7e2975d023292a48cf535e1f71e311b7d9e3996 WatchSource:0}: Error finding container c19dc2dc0751df0cf8402b75e7e2975d023292a48cf535e1f71e311b7d9e3996: Status 404 returned error can't find the container with id c19dc2dc0751df0cf8402b75e7e2975d023292a48cf535e1f71e311b7d9e3996
Nov 02 18:30:49.916956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:49.916919    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd" event={"ID":"604905ae-308b-4d5f-8def-7c9ecbf28d41","Type":"ContainerStarted","Data":"58eedb232cdc7c33729db8e79c8fa569fc7a3b0ebada495a2878613dd5665dee"}
Nov 02 18:30:49.917447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:49.916978    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd" event={"ID":"604905ae-308b-4d5f-8def-7c9ecbf28d41","Type":"ContainerStarted","Data":"c19dc2dc0751df0cf8402b75e7e2975d023292a48cf535e1f71e311b7d9e3996"}
Nov 02 18:30:49.936217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:49.936155    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd" podStartSLOduration=1.9361387429999999 podStartE2EDuration="1.936138743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:49.934069026 +0000 UTC m=+3462.513478433" watchObservedRunningTime="2026-11-02 18:30:49.936138743 +0000 UTC m=+3462.515548128"
Nov 02 18:30:50.093292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.093258    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:50.138399 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.138348    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca\") pod \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") "
Nov 02 18:30:50.138627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.138414    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz\" (UniqueName: \"kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz\") pod \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") "
Nov 02 18:30:50.138627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.138544    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca\") pod \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\" (UID: \"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83\") "
Nov 02 18:30:50.139135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.139104    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca" pod "c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" (UID: "c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:50.141472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.141445    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca" pod "c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" (UID: "c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:50.141627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.141498    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz" pod "c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" (UID: "c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83"). InnerVolumeSpecName "kube-api-access-2fmpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:50.240296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.240258    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:50.240296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.240294    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fmpz\" (UniqueName: \"kubernetes.io/projected/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-kube-api-access-2fmpz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:50.240296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.240306    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:50.929332 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.929298    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"
Nov 02 18:30:50.929922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.929323    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6" event={"ID":"c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83","Type":"ContainerDied","Data":"cd84a729f6b31fbc0ff729ff673413bc6b9383faf8a9aa456a90fb86e68cc739"}
Nov 02 18:30:50.929922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.929395    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd84a729f6b31fbc0ff729ff673413bc6b9383faf8a9aa456a90fb86e68cc739"
Nov 02 18:30:50.931705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.931667    5042 generic.go:356] "Generic (PLEG): container finished" podID="604905ae-308b-4d5f-8def-7c9ecbf28d41" containerID="58eedb232cdc7c33729db8e79c8fa569fc7a3b0ebada495a2878613dd5665dee" exitCode=0
Nov 02 18:30:50.931818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:50.931743    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd" event={"ID":"604905ae-308b-4d5f-8def-7c9ecbf28d41","Type":"ContainerDied","Data":"58eedb232cdc7c33729db8e79c8fa569fc7a3b0ebada495a2878613dd5665dee"}
Nov 02 18:30:52.120061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.120023    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:52.168041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.167555    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca\") pod \"604905ae-308b-4d5f-8def-7c9ecbf28d41\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") "
Nov 02 18:30:52.168041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.167718    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca\") pod \"604905ae-308b-4d5f-8def-7c9ecbf28d41\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") "
Nov 02 18:30:52.168041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.167937    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5\" (UniqueName: \"kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5\") pod \"604905ae-308b-4d5f-8def-7c9ecbf28d41\" (UID: \"604905ae-308b-4d5f-8def-7c9ecbf28d41\") "
Nov 02 18:30:52.168041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.167989    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca" pod "604905ae-308b-4d5f-8def-7c9ecbf28d41" (UID: "604905ae-308b-4d5f-8def-7c9ecbf28d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:52.168751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.168707    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:52.170272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.170242    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca" pod "604905ae-308b-4d5f-8def-7c9ecbf28d41" (UID: "604905ae-308b-4d5f-8def-7c9ecbf28d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:52.170715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.170691    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5" pod "604905ae-308b-4d5f-8def-7c9ecbf28d41" (UID: "604905ae-308b-4d5f-8def-7c9ecbf28d41"). InnerVolumeSpecName "kube-api-access-sxgj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:52.270536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.270444    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/604905ae-308b-4d5f-8def-7c9ecbf28d41-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:52.270536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.270478    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgj5\" (UniqueName: \"kubernetes.io/projected/604905ae-308b-4d5f-8def-7c9ecbf28d41-kube-api-access-sxgj5\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:52.946538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.946508    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"
Nov 02 18:30:52.946739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.946509    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd" event={"ID":"604905ae-308b-4d5f-8def-7c9ecbf28d41","Type":"ContainerDied","Data":"c19dc2dc0751df0cf8402b75e7e2975d023292a48cf535e1f71e311b7d9e3996"}
Nov 02 18:30:52.946739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:52.946633    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19dc2dc0751df0cf8402b75e7e2975d023292a48cf535e1f71e311b7d9e3996"
Nov 02 18:30:54.003443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.003392    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main" containerID="cri-o://47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28" gracePeriod=45
Nov 02 18:30:54.035680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.035637    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main" containerID="cri-o://08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef" gracePeriod=45
Nov 02 18:30:54.344312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.344280    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:30:54.392803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.385224    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403516    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm\") pod \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403604    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs\") pod \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403625    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq\" (UniqueName: \"kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq\") pod \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403700    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm\") pod \"9126f0ac-686e-4dce-8309-31c7cb08350c\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403749    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home\") pod \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403783    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir\") pod \"9126f0ac-686e-4dce-8309-31c7cb08350c\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403890    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache\") pod \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403930    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn\" (UniqueName: \"kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn\") pod \"9126f0ac-686e-4dce-8309-31c7cb08350c\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.403985    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache\") pod \"9126f0ac-686e-4dce-8309-31c7cb08350c\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.404022    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir\") pod \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\" (UID: \"3973d45d-c19f-4da3-9795-ce61ad4f9fd4\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.404062    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs\") pod \"9126f0ac-686e-4dce-8309-31c7cb08350c\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.404099    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home\") pod \"9126f0ac-686e-4dce-8309-31c7cb08350c\" (UID: \"9126f0ac-686e-4dce-8309-31c7cb08350c\") "
Nov 02 18:30:54.407569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.404712    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home" pod "9126f0ac-686e-4dce-8309-31c7cb08350c" (UID: "9126f0ac-686e-4dce-8309-31c7cb08350c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.408244    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs" pod "3973d45d-c19f-4da3-9795-ce61ad4f9fd4" (UID: "3973d45d-c19f-4da3-9795-ce61ad4f9fd4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.413816    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home" pod "3973d45d-c19f-4da3-9795-ce61ad4f9fd4" (UID: "3973d45d-c19f-4da3-9795-ce61ad4f9fd4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.414192    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs" pod "9126f0ac-686e-4dce-8309-31c7cb08350c" (UID: "9126f0ac-686e-4dce-8309-31c7cb08350c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.417266    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache" pod "9126f0ac-686e-4dce-8309-31c7cb08350c" (UID: "9126f0ac-686e-4dce-8309-31c7cb08350c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.417637    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir" pod "9126f0ac-686e-4dce-8309-31c7cb08350c" (UID: "9126f0ac-686e-4dce-8309-31c7cb08350c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.417872    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache" pod "3973d45d-c19f-4da3-9795-ce61ad4f9fd4" (UID: "3973d45d-c19f-4da3-9795-ce61ad4f9fd4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.423756    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn" pod "9126f0ac-686e-4dce-8309-31c7cb08350c" (UID: "9126f0ac-686e-4dce-8309-31c7cb08350c"). InnerVolumeSpecName "kube-api-access-4lgdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.425145    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir" pod "3973d45d-c19f-4da3-9795-ce61ad4f9fd4" (UID: "3973d45d-c19f-4da3-9795-ce61ad4f9fd4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.425568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.425417    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq" pod "3973d45d-c19f-4da3-9795-ce61ad4f9fd4" (UID: "3973d45d-c19f-4da3-9795-ce61ad4f9fd4"). InnerVolumeSpecName "kube-api-access-5dxxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:54.426091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.425762    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm" pod "9126f0ac-686e-4dce-8309-31c7cb08350c" (UID: "9126f0ac-686e-4dce-8309-31c7cb08350c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.426091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.425786    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm" pod "3973d45d-c19f-4da3-9795-ce61ad4f9fd4" (UID: "3973d45d-c19f-4da3-9795-ce61ad4f9fd4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:30:54.506321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506216    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506263    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506281    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lgdn\" (UniqueName: \"kubernetes.io/projected/9126f0ac-686e-4dce-8309-31c7cb08350c-kube-api-access-4lgdn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506297    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506312    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506325    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9126f0ac-686e-4dce-8309-31c7cb08350c-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506338    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506350    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506361    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506374    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxxq\" (UniqueName: \"kubernetes.io/projected/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-kube-api-access-5dxxq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506387    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9126f0ac-686e-4dce-8309-31c7cb08350c-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.506800 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.506400    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3973d45d-c19f-4da3-9795-ce61ad4f9fd4-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:30:54.966006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.965971    5042 generic.go:356] "Generic (PLEG): container finished" podID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerID="08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef" exitCode=0
Nov 02 18:30:54.966901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.966026    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"
Nov 02 18:30:54.966901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.966063    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" event={"ID":"3973d45d-c19f-4da3-9795-ce61ad4f9fd4","Type":"ContainerDied","Data":"08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef"}
Nov 02 18:30:54.966901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.966111    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8" event={"ID":"3973d45d-c19f-4da3-9795-ce61ad4f9fd4","Type":"ContainerDied","Data":"0ebe44f513e7ebb44b20d5b4c54c8837eb859ff91b8e87208ecebcf9e74e4f48"}
Nov 02 18:30:54.966901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.966134    5042 scope.go:122] "RemoveContainer" containerID="08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef"
Nov 02 18:30:54.968129 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.968098    5042 generic.go:356] "Generic (PLEG): container finished" podID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerID="47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28" exitCode=0
Nov 02 18:30:54.968349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.968171    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"
Nov 02 18:30:54.968349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.968185    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" event={"ID":"9126f0ac-686e-4dce-8309-31c7cb08350c","Type":"ContainerDied","Data":"47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28"}
Nov 02 18:30:54.968349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.968241    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb" event={"ID":"9126f0ac-686e-4dce-8309-31c7cb08350c","Type":"ContainerDied","Data":"eb18ccf2e68ff58b2ca225fb83d1ea17b5c527e44e9f4a4e4183fc069c469b81"}
Nov 02 18:30:54.983513 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.983484    5042 scope.go:122] "RemoveContainer" containerID="08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef"
Nov 02 18:30:54.983986 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:54.983954    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef\": container with ID starting with 08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef not found: ID does not exist" containerID="08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef"
Nov 02 18:30:54.984117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.983998    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef"} err="failed to get container status \"08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef\": rpc error: code = NotFound desc = could not find container \"08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef\": container with ID starting with 08fe184bdd0301bcf53fb593b744e809d629c753be738c7823d37311570434ef not found: ID does not exist"
Nov 02 18:30:54.984117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:54.984029    5042 scope.go:122] "RemoveContainer" containerID="47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28"
Nov 02 18:30:55.000765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.000726    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"]
Nov 02 18:30:55.000944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.000924    5042 scope.go:122] "RemoveContainer" containerID="47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28"
Nov 02 18:30:55.001320 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:30:55.001294    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28\": container with ID starting with 47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28 not found: ID does not exist" containerID="47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28"
Nov 02 18:30:55.001418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.001329    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28"} err="failed to get container status \"47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28\": rpc error: code = NotFound desc = could not find container \"47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28\": container with ID starting with 47de51e985ea83f7ef8e55e72945506139d1441a886a6226de68ffc0c6d27d28 not found: ID does not exist"
Nov 02 18:30:55.007300 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.007268    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/distinct-2-w2-66eed1-kserve-59945c9d57-tmzwb"]
Nov 02 18:30:55.017456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.017411    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"]
Nov 02 18:30:55.021019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.020989    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-66eed1/simulator-w2-66eed1-kserve-85486cc46b-mthb8"]
Nov 02 18:30:55.544438 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.544399    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" path="/var/lib/kubelet/pods/3973d45d-c19f-4da3-9795-ce61ad4f9fd4/volumes"
Nov 02 18:30:55.544907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:55.544887    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" path="/var/lib/kubelet/pods/9126f0ac-686e-4dce-8309-31c7cb08350c/volumes"
Nov 02 18:30:58.968967 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.968919    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"]
Nov 02 18:30:58.969530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969514    5042 cpu_manager.go:414] "containerMap: removing container" podUID="604905ae-308b-4d5f-8def-7c9ecbf28d41" containerName="revoke-keys"
Nov 02 18:30:58.969577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969531    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604905ae-308b-4d5f-8def-7c9ecbf28d41" containerName="revoke-keys"
Nov 02 18:30:58.969577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969551    5042 cpu_manager.go:414] "containerMap: removing container" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main"
Nov 02 18:30:58.969577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969556    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main"
Nov 02 18:30:58.969577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969566    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" containerName="revoke-keys"
Nov 02 18:30:58.969577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969570    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" containerName="revoke-keys"
Nov 02 18:30:58.969744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969580    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main"
Nov 02 18:30:58.969744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969585    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main"
Nov 02 18:30:58.969744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969675    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9126f0ac-686e-4dce-8309-31c7cb08350c" containerName="main"
Nov 02 18:30:58.969744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969691    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="604905ae-308b-4d5f-8def-7c9ecbf28d41" containerName="revoke-keys"
Nov 02 18:30:58.969744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969700    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="3973d45d-c19f-4da3-9795-ce61ad4f9fd4" containerName="main"
Nov 02 18:30:58.969744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.969705    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" containerName="revoke-keys"
Nov 02 18:30:58.975235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.975161    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:58.980469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.979738    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:30:58.980469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.980357    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"]
Nov 02 18:30:58.980469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:58.980405    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:30:59.095499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.095457    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf9xq\" (UniqueName: \"kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.095698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.095551    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.095698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.095633    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.196983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.196931    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf9xq\" (UniqueName: \"kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.197200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.197027    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.197200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.197103    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.198002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.197929    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.200361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.200330    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.205561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.205520    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf9xq\" (UniqueName: \"kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq\") pod \"maas-api-revoke-sub-c8bc5e9e-wb2wz\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.294569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.294467    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:30:59.580120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:30:59.580086    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"]
Nov 02 18:30:59.580791 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:30:59.580748    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf7aa2b_f16a_48c6_8df1_44b8a115d530.slice/crio-886bafcc2de13185a205837613ee493f06680d8c9f0ba4451e210ab45ac66099 WatchSource:0}: Error finding container 886bafcc2de13185a205837613ee493f06680d8c9f0ba4451e210ab45ac66099: Status 404 returned error can't find the container with id 886bafcc2de13185a205837613ee493f06680d8c9f0ba4451e210ab45ac66099
Nov 02 18:31:00.026715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:00.026672    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz" event={"ID":"abf7aa2b-f16a-48c6-8df1-44b8a115d530","Type":"ContainerStarted","Data":"906386577c6f3ee86ff6a3770c401ee30c1b5e90b9080bd7d3a03e56ecf0ea77"}
Nov 02 18:31:00.027163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:00.026781    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz" event={"ID":"abf7aa2b-f16a-48c6-8df1-44b8a115d530","Type":"ContainerStarted","Data":"886bafcc2de13185a205837613ee493f06680d8c9f0ba4451e210ab45ac66099"}
Nov 02 18:31:00.044622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:00.044560    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz" podStartSLOduration=2.044539508 podStartE2EDuration="2.044539508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:31:00.043066634 +0000 UTC m=+3472.622476017" watchObservedRunningTime="2026-11-02 18:31:00.044539508 +0000 UTC m=+3472.623948895"
Nov 02 18:31:01.036367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:01.036332    5042 generic.go:356] "Generic (PLEG): container finished" podID="abf7aa2b-f16a-48c6-8df1-44b8a115d530" containerID="906386577c6f3ee86ff6a3770c401ee30c1b5e90b9080bd7d3a03e56ecf0ea77" exitCode=0
Nov 02 18:31:01.036832 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:01.036419    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz" event={"ID":"abf7aa2b-f16a-48c6-8df1-44b8a115d530","Type":"ContainerDied","Data":"906386577c6f3ee86ff6a3770c401ee30c1b5e90b9080bd7d3a03e56ecf0ea77"}
Nov 02 18:31:02.240394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.240368    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:31:02.358684 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.358573    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca\") pod \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") "
Nov 02 18:31:02.358923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.358727    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq\" (UniqueName: \"kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq\") pod \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") "
Nov 02 18:31:02.358923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.358781    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca\") pod \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\" (UID: \"abf7aa2b-f16a-48c6-8df1-44b8a115d530\") "
Nov 02 18:31:02.359172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.359146    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca" pod "abf7aa2b-f16a-48c6-8df1-44b8a115d530" (UID: "abf7aa2b-f16a-48c6-8df1-44b8a115d530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:31:02.359312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.359298    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:02.361113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.361087    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca" pod "abf7aa2b-f16a-48c6-8df1-44b8a115d530" (UID: "abf7aa2b-f16a-48c6-8df1-44b8a115d530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:31:02.361194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.361102    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq" pod "abf7aa2b-f16a-48c6-8df1-44b8a115d530" (UID: "abf7aa2b-f16a-48c6-8df1-44b8a115d530"). InnerVolumeSpecName "kube-api-access-lf9xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:02.460535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.460493    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf9xq\" (UniqueName: \"kubernetes.io/projected/abf7aa2b-f16a-48c6-8df1-44b8a115d530-kube-api-access-lf9xq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:02.460535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:02.460534    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf7aa2b-f16a-48c6-8df1-44b8a115d530-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:03.059062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:03.059030    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"
Nov 02 18:31:03.059264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:03.059059    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz" event={"ID":"abf7aa2b-f16a-48c6-8df1-44b8a115d530","Type":"ContainerDied","Data":"886bafcc2de13185a205837613ee493f06680d8c9f0ba4451e210ab45ac66099"}
Nov 02 18:31:03.059264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:03.059114    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886bafcc2de13185a205837613ee493f06680d8c9f0ba4451e210ab45ac66099"
Nov 02 18:31:09.026933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:09.026894    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"]
Nov 02 18:31:09.032284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:09.032249    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e70e438-jmcww"]
Nov 02 18:31:09.545437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:09.545401    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b075f3-7946-46f3-b9bb-b0b479bafa3c" path="/var/lib/kubelet/pods/33b075f3-7946-46f3-b9bb-b0b479bafa3c/volumes"
Nov 02 18:31:09.678467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:09.678429    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"]
Nov 02 18:31:11.505190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:11.505152    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"]
Nov 02 18:31:18.157962 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:18.157918    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"]
Nov 02 18:31:18.158473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:18.158188    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" containerID="cri-o://cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec" gracePeriod=30
Nov 02 18:31:18.545643 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:31:18.545606    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:31:20.822588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:20.822538    5042 patch_prober.go:29] interesting pod/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:31:20.823139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:20.822616    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:31:23.442955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.442929    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 18:31:23.541164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541125    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541166    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541208    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541234    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541336    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541385    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541451    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541526    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.541626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541551    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg\") pod \"fcba8ae9-d213-4e57-b57c-858c149f6038\" (UID: \"fcba8ae9-d213-4e57-b57c-858c149f6038\") "
Nov 02 18:31:23.542021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.541990    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:23.542224 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.542069    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:31:23.542367 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.542167    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:23.542469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.542216    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:23.542469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.542370    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:23.544643 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.544615    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "kube-api-access-dt5xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:23.544907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.544881    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:23.545320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.544972    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:23.545320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.545062    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo" pod "fcba8ae9-d213-4e57-b57c-858c149f6038" (UID: "fcba8ae9-d213-4e57-b57c-858c149f6038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:31:23.642759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642721    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.642759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642754    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.642759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642765    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.643081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642774    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.643081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642784    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.643081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642795    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcba8ae9-d213-4e57-b57c-858c149f6038-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.643081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642809    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5xg\" (UniqueName: \"kubernetes.io/projected/fcba8ae9-d213-4e57-b57c-858c149f6038-kube-api-access-dt5xg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.643081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642819    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:23.643081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:23.642828    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcba8ae9-d213-4e57-b57c-858c149f6038-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:24.265947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.265908    5042 generic.go:356] "Generic (PLEG): container finished" podID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerID="cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec" exitCode=0
Nov 02 18:31:24.266198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.265969    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"
Nov 02 18:31:24.266198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.265981    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" event={"ID":"fcba8ae9-d213-4e57-b57c-858c149f6038","Type":"ContainerDied","Data":"cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec"}
Nov 02 18:31:24.266198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.266042    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7" event={"ID":"fcba8ae9-d213-4e57-b57c-858c149f6038","Type":"ContainerDied","Data":"ac08408e0c94e4a221572ba97d4a612d175a1a8f4f2988ef848d139838f2e03a"}
Nov 02 18:31:24.266198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.266052    5042 scope.go:122] "RemoveContainer" containerID="cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec"
Nov 02 18:31:24.278987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.278778    5042 scope.go:122] "RemoveContainer" containerID="cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec"
Nov 02 18:31:24.279175 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:31:24.279145    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec\": container with ID starting with cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec not found: ID does not exist" containerID="cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec"
Nov 02 18:31:24.279247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.279192    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec"} err="failed to get container status \"cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec\": rpc error: code = NotFound desc = could not find container \"cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec\": container with ID starting with cd38a9a8257131ee1b31bae832cade7e4bffc255f1a0ec64d9a18af8ae886cec not found: ID does not exist"
Nov 02 18:31:24.292774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.292740    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"]
Nov 02 18:31:24.296632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.296609    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-66eed1-openshift-default-6dbfdfb4f5-kfbd7"]
Nov 02 18:31:24.725885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:24.724083    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" podUID="fdeafd72-b1e2-478d-9aee-2ff97f377909" containerName="main" containerID="cri-o://2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67" gracePeriod=45
Nov 02 18:31:25.182762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.182734    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:31:25.188666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188620    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir\") pod \"fdeafd72-b1e2-478d-9aee-2ff97f377909\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") "
Nov 02 18:31:25.188826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188681    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm\") pod \"fdeafd72-b1e2-478d-9aee-2ff97f377909\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") "
Nov 02 18:31:25.188826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188730    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache\") pod \"fdeafd72-b1e2-478d-9aee-2ff97f377909\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") "
Nov 02 18:31:25.188931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188855    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs\" (UniqueName: \"kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs\") pod \"fdeafd72-b1e2-478d-9aee-2ff97f377909\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") "
Nov 02 18:31:25.188931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188877    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir" pod "fdeafd72-b1e2-478d-9aee-2ff97f377909" (UID: "fdeafd72-b1e2-478d-9aee-2ff97f377909"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:25.189000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188945    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf\" (UniqueName: \"kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf\") pod \"fdeafd72-b1e2-478d-9aee-2ff97f377909\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") "
Nov 02 18:31:25.189000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.188983    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home\") pod \"fdeafd72-b1e2-478d-9aee-2ff97f377909\" (UID: \"fdeafd72-b1e2-478d-9aee-2ff97f377909\") "
Nov 02 18:31:25.189526 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.189498    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:25.189781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.189759    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home" pod "fdeafd72-b1e2-478d-9aee-2ff97f377909" (UID: "fdeafd72-b1e2-478d-9aee-2ff97f377909"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:25.190047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.190020    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache" pod "fdeafd72-b1e2-478d-9aee-2ff97f377909" (UID: "fdeafd72-b1e2-478d-9aee-2ff97f377909"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:25.191796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.191764    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs" pod "fdeafd72-b1e2-478d-9aee-2ff97f377909" (UID: "fdeafd72-b1e2-478d-9aee-2ff97f377909"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:31:25.192010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.191895    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm" pod "fdeafd72-b1e2-478d-9aee-2ff97f377909" (UID: "fdeafd72-b1e2-478d-9aee-2ff97f377909"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:25.192413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.192376    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf" pod "fdeafd72-b1e2-478d-9aee-2ff97f377909" (UID: "fdeafd72-b1e2-478d-9aee-2ff97f377909"). InnerVolumeSpecName "kube-api-access-w7pnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:25.294449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.294410    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:25.294631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.294474    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:25.294631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.294489    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdeafd72-b1e2-478d-9aee-2ff97f377909-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:25.294631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.294503    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7pnf\" (UniqueName: \"kubernetes.io/projected/fdeafd72-b1e2-478d-9aee-2ff97f377909-kube-api-access-w7pnf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:25.294631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.294515    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdeafd72-b1e2-478d-9aee-2ff97f377909-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:25.300205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.300165    5042 generic.go:356] "Generic (PLEG): container finished" podID="fdeafd72-b1e2-478d-9aee-2ff97f377909" containerID="2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67" exitCode=0
Nov 02 18:31:25.300398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.300267    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" event={"ID":"fdeafd72-b1e2-478d-9aee-2ff97f377909","Type":"ContainerDied","Data":"2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67"}
Nov 02 18:31:25.300398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.300307    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" event={"ID":"fdeafd72-b1e2-478d-9aee-2ff97f377909","Type":"ContainerDied","Data":"697319c860af1bcfecca885bf4c8ddffe35090af6ea26be373f3945cda594c04"}
Nov 02 18:31:25.300398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.300329    5042 scope.go:122] "RemoveContainer" containerID="2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67"
Nov 02 18:31:25.300592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.300471    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"
Nov 02 18:31:25.326113 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.326086    5042 scope.go:122] "RemoveContainer" containerID="2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67"
Nov 02 18:31:25.326578 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:31:25.326549    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67\": container with ID starting with 2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67 not found: ID does not exist" containerID="2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67"
Nov 02 18:31:25.326764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.326735    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67"} err="failed to get container status \"2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67\": rpc error: code = NotFound desc = could not find container \"2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67\": container with ID starting with 2a85ad3f9bd220a3354985236b35d7ea23be6f47ef44ca6ddb69690ad57e4b67 not found: ID does not exist"
Nov 02 18:31:25.342348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.342314    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"]
Nov 02 18:31:25.348012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.347952    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4"]
Nov 02 18:31:25.545729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.545682    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" path="/var/lib/kubelet/pods/fcba8ae9-d213-4e57-b57c-858c149f6038/volumes"
Nov 02 18:31:25.546284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:25.546264    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdeafd72-b1e2-478d-9aee-2ff97f377909" path="/var/lib/kubelet/pods/fdeafd72-b1e2-478d-9aee-2ff97f377909/volumes"
Nov 02 18:31:26.173313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.173270    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-47e1c2/rate-test-model-47e1c2-kserve-cb766f557-mnwf4" podUID="fdeafd72-b1e2-478d-9aee-2ff97f377909" containerName="main" probeResult="failure" output="Get \"https://10.132.1.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:31:26.529999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.529956    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4" podUID="ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" containerName="main" containerID="cri-o://1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f" gracePeriod=45
Nov 02 18:31:26.812779 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.812752    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:31:26.816192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816163    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache\") pod \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") "
Nov 02 18:31:26.816344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816275    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2\" (UniqueName: \"kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2\") pod \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") "
Nov 02 18:31:26.816344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816334    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm\") pod \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") "
Nov 02 18:31:26.816474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816375    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs\") pod \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") "
Nov 02 18:31:26.816474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816458    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir\") pod \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") "
Nov 02 18:31:26.816599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816473    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache" pod "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" (UID: "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:26.816906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.816888    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:26.817279 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.817119    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir" pod "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" (UID: "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:26.819226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.819182    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm" pod "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" (UID: "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:26.819532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.819507    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2" pod "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" (UID: "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3"). InnerVolumeSpecName "kube-api-access-5pln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:26.822383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.822358    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs" pod "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" (UID: "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:31:26.917939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.917904    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home\") pod \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\" (UID: \"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3\") "
Nov 02 18:31:26.918154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.918120    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home" pod "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" (UID: "ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:26.918335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.918320    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:26.918408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.918338    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pln2\" (UniqueName: \"kubernetes.io/projected/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-kube-api-access-5pln2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:26.918408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.918347    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:26.918408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.918356    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:26.918408 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:26.918366    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:27.327081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.327044    5042 generic.go:356] "Generic (PLEG): container finished" podID="ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" containerID="1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f" exitCode=0
Nov 02 18:31:27.327608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.327101    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"
Nov 02 18:31:27.327608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.327126    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4" event={"ID":"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3","Type":"ContainerDied","Data":"1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f"}
Nov 02 18:31:27.327608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.327172    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4" event={"ID":"ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3","Type":"ContainerDied","Data":"2c294732ce51f51c54b8ab3f41fe8fcfdba6dd29f4f853862ebf70817c512e60"}
Nov 02 18:31:27.327608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.327189    5042 scope.go:122] "RemoveContainer" containerID="1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f"
Nov 02 18:31:27.338301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.338274    5042 scope.go:122] "RemoveContainer" containerID="1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f"
Nov 02 18:31:27.338706 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:31:27.338681    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f\": container with ID starting with 1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f not found: ID does not exist" containerID="1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f"
Nov 02 18:31:27.338807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.338716    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f"} err="failed to get container status \"1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f\": rpc error: code = NotFound desc = could not find container \"1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f\": container with ID starting with 1013cd8e4b5b9e044a92545b8af881974b8f5b4e3a97a0d848ba5018c960941f not found: ID does not exist"
Nov 02 18:31:27.351672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.351641    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"]
Nov 02 18:31:27.355126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.355101    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-9165c3/rate-test-model-9165c3-kserve-7b4c94c7c-q64c4"]
Nov 02 18:31:27.544710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:27.544677    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" path="/var/lib/kubelet/pods/ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3/volumes"
Nov 02 18:31:33.134571 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:33.134423    5042 scope.go:122] "RemoveContainer" containerID="c58175d164acc78afdc570038688729b1fdc4ab3ae9ad1de7fdf8a5a233030cc"
Nov 02 18:31:33.151026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:33.151005    5042 scope.go:122] "RemoveContainer" containerID="7ecf841c1f053769ff5a59979dbe1738e49545f284b1a11ed7b358633901ad51"
Nov 02 18:31:33.161350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:33.161327    5042 scope.go:122] "RemoveContainer" containerID="b819e0f255bed8c35d4a8521f44668c59ed8300bc91618275a1180933ca052d6"
Nov 02 18:31:48.538660 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:31:48.538621    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:31:53.883159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883121    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"]
Nov 02 18:31:53.883675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883661    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy"
Nov 02 18:31:53.883675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883675    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy"
Nov 02 18:31:53.883758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883698    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" containerName="main"
Nov 02 18:31:53.883758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883703    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" containerName="main"
Nov 02 18:31:53.883758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883714    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fdeafd72-b1e2-478d-9aee-2ff97f377909" containerName="main"
Nov 02 18:31:53.883758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883722    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdeafd72-b1e2-478d-9aee-2ff97f377909" containerName="main"
Nov 02 18:31:53.883758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883742    5042 cpu_manager.go:414] "containerMap: removing container" podUID="abf7aa2b-f16a-48c6-8df1-44b8a115d530" containerName="revoke-keys"
Nov 02 18:31:53.883758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883746    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf7aa2b-f16a-48c6-8df1-44b8a115d530" containerName="revoke-keys"
Nov 02 18:31:53.884021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883831    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcba8ae9-d213-4e57-b57c-858c149f6038" containerName="istio-proxy"
Nov 02 18:31:53.884021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883859    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed38f5f4-07bf-4fe1-bd65-3cb56a8106b3" containerName="main"
Nov 02 18:31:53.884021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883869    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdeafd72-b1e2-478d-9aee-2ff97f377909" containerName="main"
Nov 02 18:31:53.884021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.883884    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf7aa2b-f16a-48c6-8df1-44b8a115d530" containerName="revoke-keys"
Nov 02 18:31:53.887481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.887462    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:53.895924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.895884    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"]
Nov 02 18:31:53.896976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.896954    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:31:53.897171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:53.897151    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:31:54.014895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.014855    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.015119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.014916    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.015119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.015002    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f682f\" (UniqueName: \"kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.116388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.116347    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.116616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.116398    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.116616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.116423    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f682f\" (UniqueName: \"kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.117289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.117262    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.119086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.119061    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.124692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.124660    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f682f\" (UniqueName: \"kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f\") pod \"maas-api-revoke-sub-0f33f80c-94nxm\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.213468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.213435    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:54.496115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.496085    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"]
Nov 02 18:31:54.496966 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:31:54.496941    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fa48a8_767e_434d_abb1_f99968544a75.slice/crio-c6d7f09793e6619fe04e58619c9b0729e281ea98e624899076533da33d8815a6 WatchSource:0}: Error finding container c6d7f09793e6619fe04e58619c9b0729e281ea98e624899076533da33d8815a6: Status 404 returned error can't find the container with id c6d7f09793e6619fe04e58619c9b0729e281ea98e624899076533da33d8815a6
Nov 02 18:31:54.527687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:54.527649    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm" event={"ID":"24fa48a8-767e-434d-abb1-f99968544a75","Type":"ContainerStarted","Data":"c6d7f09793e6619fe04e58619c9b0729e281ea98e624899076533da33d8815a6"}
Nov 02 18:31:55.539682 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:55.539652    5042 generic.go:356] "Generic (PLEG): container finished" podID="24fa48a8-767e-434d-abb1-f99968544a75" containerID="2a9571945fd6019fc44d87e71de44bc5aa612169db5a22b058415cf9e68d4e14" exitCode=0
Nov 02 18:31:55.544982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:55.544949    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm" event={"ID":"24fa48a8-767e-434d-abb1-f99968544a75","Type":"ContainerDied","Data":"2a9571945fd6019fc44d87e71de44bc5aa612169db5a22b058415cf9e68d4e14"}
Nov 02 18:31:56.186893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.186854    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"]
Nov 02 18:31:56.187174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.187119    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy" containerID="cri-o://c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf" gracePeriod=30
Nov 02 18:31:56.711964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.711934    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:56.848632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.848546    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca\") pod \"24fa48a8-767e-434d-abb1-f99968544a75\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") "
Nov 02 18:31:56.848809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.848659    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f\" (UniqueName: \"kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f\") pod \"24fa48a8-767e-434d-abb1-f99968544a75\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") "
Nov 02 18:31:56.848809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.848684    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca\") pod \"24fa48a8-767e-434d-abb1-f99968544a75\" (UID: \"24fa48a8-767e-434d-abb1-f99968544a75\") "
Nov 02 18:31:56.849135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.849113    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca" pod "24fa48a8-767e-434d-abb1-f99968544a75" (UID: "24fa48a8-767e-434d-abb1-f99968544a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:31:56.850917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.850891    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca" pod "24fa48a8-767e-434d-abb1-f99968544a75" (UID: "24fa48a8-767e-434d-abb1-f99968544a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:31:56.851028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.850989    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f" pod "24fa48a8-767e-434d-abb1-f99968544a75" (UID: "24fa48a8-767e-434d-abb1-f99968544a75"). InnerVolumeSpecName "kube-api-access-f682f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:56.950048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.950013    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f682f\" (UniqueName: \"kubernetes.io/projected/24fa48a8-767e-434d-abb1-f99968544a75-kube-api-access-f682f\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:56.950048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.950047    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:56.950273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:56.950064    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24fa48a8-767e-434d-abb1-f99968544a75-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:31:57.555209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:57.555185    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"
Nov 02 18:31:57.555411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:57.555222    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm" event={"ID":"24fa48a8-767e-434d-abb1-f99968544a75","Type":"ContainerDied","Data":"c6d7f09793e6619fe04e58619c9b0729e281ea98e624899076533da33d8815a6"}
Nov 02 18:31:57.555411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:31:57.555276    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d7f09793e6619fe04e58619c9b0729e281ea98e624899076533da33d8815a6"
Nov 02 18:32:01.472066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.471999    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:32:01.584658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.584619    5042 generic.go:356] "Generic (PLEG): container finished" podID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerID="c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf" exitCode=0
Nov 02 18:32:01.584884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.584676    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"
Nov 02 18:32:01.584884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.584679    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" event={"ID":"b1ffc453-84fb-451d-892b-397ec410a2a1","Type":"ContainerDied","Data":"c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf"}
Nov 02 18:32:01.584884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.584735    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh" event={"ID":"b1ffc453-84fb-451d-892b-397ec410a2a1","Type":"ContainerDied","Data":"b3183866081ed4a07c95e2994e9d18251b32448061dede9e5a5383363dea9286"}
Nov 02 18:32:01.584884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.584767    5042 scope.go:122] "RemoveContainer" containerID="c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf"
Nov 02 18:32:01.598185 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.598164    5042 scope.go:122] "RemoveContainer" containerID="c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf"
Nov 02 18:32:01.598514 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:32:01.598486    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf\": container with ID starting with c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf not found: ID does not exist" containerID="c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf"
Nov 02 18:32:01.598575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.598529    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf"} err="failed to get container status \"c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf\": rpc error: code = NotFound desc = could not find container \"c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf\": container with ID starting with c398b425de5e77a2976b758e709415aa77b76833b843a7c20e8ed9a1582003cf not found: ID does not exist"
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.631763    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.631902    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.631970    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632278    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632368    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632472    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632529    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632629    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632719    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.632757    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token\") pod \"b1ffc453-84fb-451d-892b-397ec410a2a1\" (UID: \"b1ffc453-84fb-451d-892b-397ec410a2a1\") "
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.633062    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.633250    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.633311    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:32:01.634581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.633766    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:01.638885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.638605    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.638885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.638632    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.638885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.638646    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.638885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.638661    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1ffc453-84fb-451d-892b-397ec410a2a1-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.638885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.638675    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.639271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.639117    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:32:01.639271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.639234    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:01.641487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.641421    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:01.641643 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.641490    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645" pod "b1ffc453-84fb-451d-892b-397ec410a2a1" (UID: "b1ffc453-84fb-451d-892b-397ec410a2a1"). InnerVolumeSpecName "kube-api-access-cm645". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:01.739720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.739630    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm645\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-kube-api-access-cm645\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.739720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.739664    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.739720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.739675    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.739720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.739684    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1ffc453-84fb-451d-892b-397ec410a2a1-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:01.915136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.915095    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"]
Nov 02 18:32:01.920386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:01.920355    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-b6kzh"]
Nov 02 18:32:03.542818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:03.542775    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" path="/var/lib/kubelet/pods/b1ffc453-84fb-451d-892b-397ec410a2a1/volumes"
Nov 02 18:32:04.082878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.082818    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"]
Nov 02 18:32:04.083424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.083408    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy"
Nov 02 18:32:04.083495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.083424    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy"
Nov 02 18:32:04.083495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.083438    5042 cpu_manager.go:414] "containerMap: removing container" podUID="24fa48a8-767e-434d-abb1-f99968544a75" containerName="revoke-keys"
Nov 02 18:32:04.083495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.083443    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24fa48a8-767e-434d-abb1-f99968544a75" containerName="revoke-keys"
Nov 02 18:32:04.083727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.083553    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ffc453-84fb-451d-892b-397ec410a2a1" containerName="istio-proxy"
Nov 02 18:32:04.083727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.083565    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="24fa48a8-767e-434d-abb1-f99968544a75" containerName="revoke-keys"
Nov 02 18:32:04.088709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.088686    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.095942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.092294    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:32:04.095942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.093231    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:32:04.095942 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.094170    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"]
Nov 02 18:32:04.164850 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.164808    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsr5k\" (UniqueName: \"kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.165030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.164944    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.165030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.165002    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.265958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.265923    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsr5k\" (UniqueName: \"kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.266123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.265972    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.266123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.266018    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.266727 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.266703    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.268690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.268665    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.276103 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.276073    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsr5k\" (UniqueName: \"kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k\") pod \"maas-api-revoke-sub-326225ac-mbdrz\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.405383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.405292    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:04.688277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:04.688251    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"]
Nov 02 18:32:04.691477 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:32:04.691440    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329c5b33_9bf4_4505_a3d1_5ff44ebf248c.slice/crio-244bcb53ccf0117b933ddceaa691d758e34514ee59393e46d1a246b8d2eb407f WatchSource:0}: Error finding container 244bcb53ccf0117b933ddceaa691d758e34514ee59393e46d1a246b8d2eb407f: Status 404 returned error can't find the container with id 244bcb53ccf0117b933ddceaa691d758e34514ee59393e46d1a246b8d2eb407f
Nov 02 18:32:05.623022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:05.622989    5042 generic.go:356] "Generic (PLEG): container finished" podID="329c5b33-9bf4-4505-a3d1-5ff44ebf248c" containerID="cac8dbe9fbef993a147011f707fd14619c6f2b9511d3b6cd22fc6bac924f5181" exitCode=0
Nov 02 18:32:05.623215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:05.623075    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz" event={"ID":"329c5b33-9bf4-4505-a3d1-5ff44ebf248c","Type":"ContainerDied","Data":"cac8dbe9fbef993a147011f707fd14619c6f2b9511d3b6cd22fc6bac924f5181"}
Nov 02 18:32:05.623215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:05.623133    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz" event={"ID":"329c5b33-9bf4-4505-a3d1-5ff44ebf248c","Type":"ContainerStarted","Data":"244bcb53ccf0117b933ddceaa691d758e34514ee59393e46d1a246b8d2eb407f"}
Nov 02 18:32:06.783616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.783588    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:06.919478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.919384    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca\") pod \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") "
Nov 02 18:32:06.919478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.919422    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k\" (UniqueName: \"kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k\") pod \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") "
Nov 02 18:32:06.919739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.919541    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca\") pod \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\" (UID: \"329c5b33-9bf4-4505-a3d1-5ff44ebf248c\") "
Nov 02 18:32:06.919994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.919963    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca" pod "329c5b33-9bf4-4505-a3d1-5ff44ebf248c" (UID: "329c5b33-9bf4-4505-a3d1-5ff44ebf248c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:32:06.921794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.921768    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca" pod "329c5b33-9bf4-4505-a3d1-5ff44ebf248c" (UID: "329c5b33-9bf4-4505-a3d1-5ff44ebf248c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:32:06.921926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:06.921899    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k" pod "329c5b33-9bf4-4505-a3d1-5ff44ebf248c" (UID: "329c5b33-9bf4-4505-a3d1-5ff44ebf248c"). InnerVolumeSpecName "kube-api-access-gsr5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:07.020507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:07.020467    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:07.020507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:07.020503    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsr5k\" (UniqueName: \"kubernetes.io/projected/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-kube-api-access-gsr5k\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:07.020507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:07.020513    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329c5b33-9bf4-4505-a3d1-5ff44ebf248c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:07.644833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:07.644803    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"
Nov 02 18:32:07.644833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:07.644798    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz" event={"ID":"329c5b33-9bf4-4505-a3d1-5ff44ebf248c","Type":"ContainerDied","Data":"244bcb53ccf0117b933ddceaa691d758e34514ee59393e46d1a246b8d2eb407f"}
Nov 02 18:32:07.645131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:07.644911    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="244bcb53ccf0117b933ddceaa691d758e34514ee59393e46d1a246b8d2eb407f"
Nov 02 18:32:14.774910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.774874    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"]
Nov 02 18:32:14.775447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.775433    5042 cpu_manager.go:414] "containerMap: removing container" podUID="329c5b33-9bf4-4505-a3d1-5ff44ebf248c" containerName="revoke-keys"
Nov 02 18:32:14.775491 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.775449    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329c5b33-9bf4-4505-a3d1-5ff44ebf248c" containerName="revoke-keys"
Nov 02 18:32:14.775535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.775526    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="329c5b33-9bf4-4505-a3d1-5ff44ebf248c" containerName="revoke-keys"
Nov 02 18:32:14.779214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.779194    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:14.784347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.784263    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:32:14.784347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.784295    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:32:14.790340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.790313    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"]
Nov 02 18:32:14.938591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.938558    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:14.938784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.938753    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:14.938925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:14.938908    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvkt\" (UniqueName: \"kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.039757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.039660    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.039757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.039726    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvkt\" (UniqueName: \"kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.040036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.039801    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.040358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.040335    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.042380 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.042352    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.048357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.048324    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvkt\" (UniqueName: \"kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt\") pod \"maas-api-revoke-sub-32cc49b1-4f67x\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.106219 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.106166    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:15.391316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.391284    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"]
Nov 02 18:32:15.392272 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:32:15.392231    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c92a41f_62c2_47dd_9d80_4a34e0edbc81.slice/crio-4cbb226930d12b9c90ab02f489488add30a6e28bf9f83f89818c9bc2853d8a3a WatchSource:0}: Error finding container 4cbb226930d12b9c90ab02f489488add30a6e28bf9f83f89818c9bc2853d8a3a: Status 404 returned error can't find the container with id 4cbb226930d12b9c90ab02f489488add30a6e28bf9f83f89818c9bc2853d8a3a
Nov 02 18:32:15.702420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:15.702385    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x" event={"ID":"1c92a41f-62c2-47dd-9d80-4a34e0edbc81","Type":"ContainerStarted","Data":"4cbb226930d12b9c90ab02f489488add30a6e28bf9f83f89818c9bc2853d8a3a"}
Nov 02 18:32:16.712215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:16.712175    5042 generic.go:356] "Generic (PLEG): container finished" podID="1c92a41f-62c2-47dd-9d80-4a34e0edbc81" containerID="f443d08b0cbd0f00467c5ff57e81039b869c15d25af152f32af0099c809dbd8f" exitCode=0
Nov 02 18:32:16.712656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:16.712272    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x" event={"ID":"1c92a41f-62c2-47dd-9d80-4a34e0edbc81","Type":"ContainerDied","Data":"f443d08b0cbd0f00467c5ff57e81039b869c15d25af152f32af0099c809dbd8f"}
Nov 02 18:32:17.893240 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:17.893214    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:18.081898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.081799    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca\") pod \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") "
Nov 02 18:32:18.082048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.081900    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt\" (UniqueName: \"kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt\") pod \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") "
Nov 02 18:32:18.082149 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.082130    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca\") pod \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\" (UID: \"1c92a41f-62c2-47dd-9d80-4a34e0edbc81\") "
Nov 02 18:32:18.082194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.082164    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca" pod "1c92a41f-62c2-47dd-9d80-4a34e0edbc81" (UID: "1c92a41f-62c2-47dd-9d80-4a34e0edbc81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:32:18.082552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.082526    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:18.084242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.084213    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt" pod "1c92a41f-62c2-47dd-9d80-4a34e0edbc81" (UID: "1c92a41f-62c2-47dd-9d80-4a34e0edbc81"). InnerVolumeSpecName "kube-api-access-svvkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:18.084495 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.084461    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca" pod "1c92a41f-62c2-47dd-9d80-4a34e0edbc81" (UID: "1c92a41f-62c2-47dd-9d80-4a34e0edbc81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:32:18.184027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.183993    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:18.184027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.184024    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvkt\" (UniqueName: \"kubernetes.io/projected/1c92a41f-62c2-47dd-9d80-4a34e0edbc81-kube-api-access-svvkt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:18.538901 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:32:18.538829    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:32:18.730049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.730021    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"
Nov 02 18:32:18.730221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.730056    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x" event={"ID":"1c92a41f-62c2-47dd-9d80-4a34e0edbc81","Type":"ContainerDied","Data":"4cbb226930d12b9c90ab02f489488add30a6e28bf9f83f89818c9bc2853d8a3a"}
Nov 02 18:32:18.730221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:18.730108    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cbb226930d12b9c90ab02f489488add30a6e28bf9f83f89818c9bc2853d8a3a"
Nov 02 18:32:25.292823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.292781    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"]
Nov 02 18:32:25.293666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.293644    5042 cpu_manager.go:414] "containerMap: removing container" podUID="1c92a41f-62c2-47dd-9d80-4a34e0edbc81" containerName="revoke-keys"
Nov 02 18:32:25.293729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.293669    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c92a41f-62c2-47dd-9d80-4a34e0edbc81" containerName="revoke-keys"
Nov 02 18:32:25.293862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.293825    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c92a41f-62c2-47dd-9d80-4a34e0edbc81" containerName="revoke-keys"
Nov 02 18:32:25.298027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.298003    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.302943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.302875    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"]
Nov 02 18:32:25.306039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.306017    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:32:25.306197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.306033    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:32:25.370194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.370156    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lg2v\" (UniqueName: \"kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.370428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.370307    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.370522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.370454    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.471444 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.471405    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.471624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.471457    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.471624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.471540    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lg2v\" (UniqueName: \"kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.472187 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.472158    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.474021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.474002    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.479362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.479342    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lg2v\" (UniqueName: \"kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v\") pod \"maas-api-revoke-sub-da187c9c-m4k9g\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.617351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.617275    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:25.899530 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:25.899507    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"]
Nov 02 18:32:25.900768 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:32:25.900742    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c9377c_f1e2_457d_95f8_2e26a8463ef6.slice/crio-7e2e32f5e83dcc6438f194cd708cc376b6a111427b2143c00622a553b52f699e WatchSource:0}: Error finding container 7e2e32f5e83dcc6438f194cd708cc376b6a111427b2143c00622a553b52f699e: Status 404 returned error can't find the container with id 7e2e32f5e83dcc6438f194cd708cc376b6a111427b2143c00622a553b52f699e
Nov 02 18:32:26.794917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:26.794881    5042 generic.go:356] "Generic (PLEG): container finished" podID="f0c9377c-f1e2-457d-95f8-2e26a8463ef6" containerID="bd007dc5f69135c604d7bbf964d415a28a5a418877f6fcfdb07e336a6583dc72" exitCode=0
Nov 02 18:32:26.795272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:26.794939    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g" event={"ID":"f0c9377c-f1e2-457d-95f8-2e26a8463ef6","Type":"ContainerDied","Data":"bd007dc5f69135c604d7bbf964d415a28a5a418877f6fcfdb07e336a6583dc72"}
Nov 02 18:32:26.795272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:26.794990    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g" event={"ID":"f0c9377c-f1e2-457d-95f8-2e26a8463ef6","Type":"ContainerStarted","Data":"7e2e32f5e83dcc6438f194cd708cc376b6a111427b2143c00622a553b52f699e"}
Nov 02 18:32:27.029065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:27.028959    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"]
Nov 02 18:32:27.033599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:27.033563    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-491e8b2b-4k8zv"]
Nov 02 18:32:27.543904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:27.543860    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7caf1ff8-8099-40b9-aca9-0c2b70330b89" path="/var/lib/kubelet/pods/7caf1ff8-8099-40b9-aca9-0c2b70330b89/volumes"
Nov 02 18:32:27.970358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:27.970326    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:28.008959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.008921    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v\" (UniqueName: \"kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v\") pod \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") "
Nov 02 18:32:28.009210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.009189    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca\") pod \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") "
Nov 02 18:32:28.009309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.009282    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca\") pod \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\" (UID: \"f0c9377c-f1e2-457d-95f8-2e26a8463ef6\") "
Nov 02 18:32:28.010176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.010145    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca" pod "f0c9377c-f1e2-457d-95f8-2e26a8463ef6" (UID: "f0c9377c-f1e2-457d-95f8-2e26a8463ef6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:32:28.011545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.011517    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca" pod "f0c9377c-f1e2-457d-95f8-2e26a8463ef6" (UID: "f0c9377c-f1e2-457d-95f8-2e26a8463ef6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:32:28.011944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.011916    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v" pod "f0c9377c-f1e2-457d-95f8-2e26a8463ef6" (UID: "f0c9377c-f1e2-457d-95f8-2e26a8463ef6"). InnerVolumeSpecName "kube-api-access-7lg2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:28.110744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.110663    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lg2v\" (UniqueName: \"kubernetes.io/projected/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-kube-api-access-7lg2v\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:28.110744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.110693    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:28.110744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.110703    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c9377c-f1e2-457d-95f8-2e26a8463ef6-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:28.814458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.814428    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"
Nov 02 18:32:28.814735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.814424    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g" event={"ID":"f0c9377c-f1e2-457d-95f8-2e26a8463ef6","Type":"ContainerDied","Data":"7e2e32f5e83dcc6438f194cd708cc376b6a111427b2143c00622a553b52f699e"}
Nov 02 18:32:28.814735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:28.814553    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2e32f5e83dcc6438f194cd708cc376b6a111427b2143c00622a553b52f699e"
Nov 02 18:32:33.231878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:33.231833    5042 scope.go:122] "RemoveContainer" containerID="e714ceca5bbad6538e4e050022b18ec5982ad6b4a9f954ed9d067035c6953919"
Nov 02 18:32:38.024734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:38.024697    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"]
Nov 02 18:32:38.027807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:38.027777    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b054f5a7-t7f68"]
Nov 02 18:32:39.543377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:39.543342    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1566fab0-31d1-4ca4-bed5-60a5da3b5a71" path="/var/lib/kubelet/pods/1566fab0-31d1-4ca4-bed5-60a5da3b5a71/volumes"
Nov 02 18:32:48.319862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.317266    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"]
Nov 02 18:32:48.319862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.317568    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv" podUID="0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" containerName="maas-api" containerID="cri-o://758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167" gracePeriod=30
Nov 02 18:32:48.672467 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:32:48.671799    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:32:48.756360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.756335    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:32:48.871385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.871350    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca\") pod \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") "
Nov 02 18:32:48.871600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.871452    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs\") pod \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") "
Nov 02 18:32:48.871600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.871574    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc\" (UniqueName: \"kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc\") pod \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") "
Nov 02 18:32:48.871600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.871596    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls\") pod \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\" (UID: \"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13\") "
Nov 02 18:32:48.871803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.871721    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca" pod "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" (UID: "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:32:48.872100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.872077    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:48.874089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.874064    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs" pod "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" (UID: "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:32:48.874220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.874158    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc" pod "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" (UID: "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13"). InnerVolumeSpecName "kube-api-access-v88sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:48.874595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.874575    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls" pod "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" (UID: "0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:32:48.973253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.973211    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v88sc\" (UniqueName: \"kubernetes.io/projected/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-kube-api-access-v88sc\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:48.973253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.973246    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:48.973253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.973258    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:48.996339 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.996306    5042 generic.go:356] "Generic (PLEG): container finished" podID="0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" containerID="758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167" exitCode=0
Nov 02 18:32:48.996553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.996364    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"
Nov 02 18:32:48.996553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.996395    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv" event={"ID":"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13","Type":"ContainerDied","Data":"758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167"}
Nov 02 18:32:48.996553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.996444    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv" event={"ID":"0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13","Type":"ContainerDied","Data":"61d7fbd61a7d8a5a30c4991805ee38f3c71c49f5c3b9fe8df3c9a23f94b82811"}
Nov 02 18:32:48.996553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:48.996466    5042 scope.go:122] "RemoveContainer" containerID="758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167"
Nov 02 18:32:49.013911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:49.013891    5042 scope.go:122] "RemoveContainer" containerID="758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167"
Nov 02 18:32:49.014225 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:32:49.014204    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167\": container with ID starting with 758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167 not found: ID does not exist" containerID="758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167"
Nov 02 18:32:49.014310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:49.014236    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167"} err="failed to get container status \"758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167\": rpc error: code = NotFound desc = could not find container \"758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167\": container with ID starting with 758d56735786db61f81e5dc38c689af30b407999f6c0f0d0a8a5b26024431167 not found: ID does not exist"
Nov 02 18:32:49.022824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:49.022794    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"]
Nov 02 18:32:49.025914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:49.025895    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-36a991-66b44fdc5-zc8hv"]
Nov 02 18:32:49.543460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:49.543428    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" path="/var/lib/kubelet/pods/0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13/volumes"
Nov 02 18:32:52.524878 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:52.524821    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"]
Nov 02 18:32:52.525429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:52.525102    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" containerID="cri-o://20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e" gracePeriod=30
Nov 02 18:32:56.567126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:56.567083    5042 patch_prober.go:29] interesting pod/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:32:56.567575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:56.567151    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:32:57.809652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.809571    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:32:57.812326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812297    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"]
Nov 02 18:32:57.812870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812854    5042 cpu_manager.go:414] "containerMap: removing container" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy"
Nov 02 18:32:57.812933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812873    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy"
Nov 02 18:32:57.812933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812909    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" containerName="maas-api"
Nov 02 18:32:57.812933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812916    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" containerName="maas-api"
Nov 02 18:32:57.813039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812953    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f0c9377c-f1e2-457d-95f8-2e26a8463ef6" containerName="revoke-keys"
Nov 02 18:32:57.813039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812959    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c9377c-f1e2-457d-95f8-2e26a8463ef6" containerName="revoke-keys"
Nov 02 18:32:57.813039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812968    5042 cpu_manager.go:414] "containerMap: removing container" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy"
Nov 02 18:32:57.813039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.812973    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy"
Nov 02 18:32:57.813176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.813119    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy"
Nov 02 18:32:57.813176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.813129    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df0e107-1ccd-4e19-b8b0-cdbf95ef9d13" containerName="maas-api"
Nov 02 18:32:57.813176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.813135    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c9377c-f1e2-457d-95f8-2e26a8463ef6" containerName="revoke-keys"
Nov 02 18:32:57.813176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.813141    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerName="istio-proxy"
Nov 02 18:32:57.816527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.816507    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:57.828733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.828701    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"]
Nov 02 18:32:57.841311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.839965    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:32:57.842504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.842480    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:32:57.904156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904118    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904194    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904285    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904384    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904443    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904490    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904526    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904580    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.904696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904618    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket\") pod \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\" (UID: \"aa2a4865-484f-4b9c-9b3a-9b340885ab7f\") "
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904982    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.904997    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905022    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrw2g\" (UniqueName: \"kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905151    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905220    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905020    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905258    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:57.905292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905239    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:57.906076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.905627    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:32:57.907722 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.907683    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "kube-api-access-cl8mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:57.907929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.907812    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:32:57.907929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.907829    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:32:57.908278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:57.908261    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy" pod "aa2a4865-484f-4b9c-9b3a-9b340885ab7f" (UID: "aa2a4865-484f-4b9c-9b3a-9b340885ab7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:32:58.006951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.006900    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.007153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007125    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.007195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007155    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrw2g\" (UniqueName: \"kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.007239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007220    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007388 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007366    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007399    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007419    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007437    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007454    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl8mb\" (UniqueName: \"kubernetes.io/projected/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-kube-api-access-cl8mb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007469    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007484    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2a4865-484f-4b9c-9b3a-9b340885ab7f-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:32:58.007692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.007666    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.009795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.009771    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.016555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.016529    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrw2g\" (UniqueName: \"kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g\") pod \"maas-api-revoke-sub-6b25bb23-qtvvp\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.083528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.083437    5042 generic.go:356] "Generic (PLEG): container finished" podID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" containerID="20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e" exitCode=0
Nov 02 18:32:58.083528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.083486    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"
Nov 02 18:32:58.083528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.083515    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" event={"ID":"aa2a4865-484f-4b9c-9b3a-9b340885ab7f","Type":"ContainerDied","Data":"20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e"}
Nov 02 18:32:58.083801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.083569    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k" event={"ID":"aa2a4865-484f-4b9c-9b3a-9b340885ab7f","Type":"ContainerDied","Data":"fd89b47f0fc086319224987559fe3abb65267b5bba3cacc0eb484e165862ea16"}
Nov 02 18:32:58.083801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.083591    5042 scope.go:122] "RemoveContainer" containerID="20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e"
Nov 02 18:32:58.097140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.097112    5042 scope.go:122] "RemoveContainer" containerID="19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"
Nov 02 18:32:58.112324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.112294    5042 scope.go:122] "RemoveContainer" containerID="20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e"
Nov 02 18:32:58.112760 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:32:58.112731    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e\": container with ID starting with 20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e not found: ID does not exist" containerID="20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e"
Nov 02 18:32:58.112917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.112777    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e"} err="failed to get container status \"20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e\": rpc error: code = NotFound desc = could not find container \"20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e\": container with ID starting with 20a646d0902609127aa002cf514da96671c83f685179dc4e3e92f950ba6c6f7e not found: ID does not exist"
Nov 02 18:32:58.112917 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.112806    5042 scope.go:122] "RemoveContainer" containerID="19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"
Nov 02 18:32:58.113453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.113422    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"]
Nov 02 18:32:58.114203 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:32:58.114171    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0\": container with ID starting with 19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0 not found: ID does not exist" containerID="19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"
Nov 02 18:32:58.114308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.114213    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0"} err="failed to get container status \"19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0\": rpc error: code = NotFound desc = could not find container \"19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0\": container with ID starting with 19ba951411701aac8b31d8c578c60f62eb5ba5f5870e3097f2cc6419392933b0 not found: ID does not exist"
Nov 02 18:32:58.120136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.120107    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-36a991-openshift-default-66bfb884c7-h2c9k"]
Nov 02 18:32:58.140734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.140697    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:32:58.463757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:58.463726    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"]
Nov 02 18:32:58.465993 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:32:58.465953    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode521b386_8648_4cd7_94e6_e1ef84a7ac84.slice/crio-a7e6a4f6b73e57db66f86d9327e9c3ec09a3900c3e3fe1fa8808980da27d2922 WatchSource:0}: Error finding container a7e6a4f6b73e57db66f86d9327e9c3ec09a3900c3e3fe1fa8808980da27d2922: Status 404 returned error can't find the container with id a7e6a4f6b73e57db66f86d9327e9c3ec09a3900c3e3fe1fa8808980da27d2922
Nov 02 18:32:59.094116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:59.094079    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp" event={"ID":"e521b386-8648-4cd7-94e6-e1ef84a7ac84","Type":"ContainerStarted","Data":"0d3273674b75f07ff8ae03b639dc2b97579be4f394ed7adedd314351a1111c8c"}
Nov 02 18:32:59.094452 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:59.094144    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp" event={"ID":"e521b386-8648-4cd7-94e6-e1ef84a7ac84","Type":"ContainerStarted","Data":"a7e6a4f6b73e57db66f86d9327e9c3ec09a3900c3e3fe1fa8808980da27d2922"}
Nov 02 18:32:59.113695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:59.113637    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp" podStartSLOduration=2.113621374 podStartE2EDuration="2.113621374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:32:59.111284508 +0000 UTC m=+3591.690693895" watchObservedRunningTime="2026-11-02 18:32:59.113621374 +0000 UTC m=+3591.693030830"
Nov 02 18:32:59.548741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:32:59.548711    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2a4865-484f-4b9c-9b3a-9b340885ab7f" path="/var/lib/kubelet/pods/aa2a4865-484f-4b9c-9b3a-9b340885ab7f/volumes"
Nov 02 18:33:00.102376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:00.102337    5042 generic.go:356] "Generic (PLEG): container finished" podID="e521b386-8648-4cd7-94e6-e1ef84a7ac84" containerID="0d3273674b75f07ff8ae03b639dc2b97579be4f394ed7adedd314351a1111c8c" exitCode=0
Nov 02 18:33:00.102987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:00.102396    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp" event={"ID":"e521b386-8648-4cd7-94e6-e1ef84a7ac84","Type":"ContainerDied","Data":"0d3273674b75f07ff8ae03b639dc2b97579be4f394ed7adedd314351a1111c8c"}
Nov 02 18:33:01.241105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.241062    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"]
Nov 02 18:33:01.241558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.241312    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw" podUID="2254cf9a-58ee-46db-86dd-e17faaee6caf" containerName="maas-api" containerID="cri-o://7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531" gracePeriod=30
Nov 02 18:33:01.335560 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.335522    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:33:01.365258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.365169    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca\") pod \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") "
Nov 02 18:33:01.365422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.365327    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca\") pod \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") "
Nov 02 18:33:01.365422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.365390    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g\" (UniqueName: \"kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g\") pod \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\" (UID: \"e521b386-8648-4cd7-94e6-e1ef84a7ac84\") "
Nov 02 18:33:01.365953 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.365922    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca" pod "e521b386-8648-4cd7-94e6-e1ef84a7ac84" (UID: "e521b386-8648-4cd7-94e6-e1ef84a7ac84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:01.380869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.373418    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca" pod "e521b386-8648-4cd7-94e6-e1ef84a7ac84" (UID: "e521b386-8648-4cd7-94e6-e1ef84a7ac84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:01.380869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.373815    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g" pod "e521b386-8648-4cd7-94e6-e1ef84a7ac84" (UID: "e521b386-8648-4cd7-94e6-e1ef84a7ac84"). InnerVolumeSpecName "kube-api-access-wrw2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:01.469671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.467707    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrw2g\" (UniqueName: \"kubernetes.io/projected/e521b386-8648-4cd7-94e6-e1ef84a7ac84-kube-api-access-wrw2g\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:01.469671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.467749    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:01.469671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.467765    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e521b386-8648-4cd7-94e6-e1ef84a7ac84-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:01.581223 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.581193    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:33:01.670301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.670200    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca\") pod \"2254cf9a-58ee-46db-86dd-e17faaee6caf\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") "
Nov 02 18:33:01.670301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.670266    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs\") pod \"2254cf9a-58ee-46db-86dd-e17faaee6caf\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") "
Nov 02 18:33:01.670301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.670301    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p\" (UniqueName: \"kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p\") pod \"2254cf9a-58ee-46db-86dd-e17faaee6caf\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") "
Nov 02 18:33:01.670636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.670331    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls\") pod \"2254cf9a-58ee-46db-86dd-e17faaee6caf\" (UID: \"2254cf9a-58ee-46db-86dd-e17faaee6caf\") "
Nov 02 18:33:01.670700 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.670641    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca" pod "2254cf9a-58ee-46db-86dd-e17faaee6caf" (UID: "2254cf9a-58ee-46db-86dd-e17faaee6caf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:01.670774 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.670756    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2254cf9a-58ee-46db-86dd-e17faaee6caf-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:01.672715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.672686    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p" pod "2254cf9a-58ee-46db-86dd-e17faaee6caf" (UID: "2254cf9a-58ee-46db-86dd-e17faaee6caf"). InnerVolumeSpecName "kube-api-access-hvb7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:01.673123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.673097    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls" pod "2254cf9a-58ee-46db-86dd-e17faaee6caf" (UID: "2254cf9a-58ee-46db-86dd-e17faaee6caf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:01.673123 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.673109    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs" pod "2254cf9a-58ee-46db-86dd-e17faaee6caf" (UID: "2254cf9a-58ee-46db-86dd-e17faaee6caf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:01.772487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.772444    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:01.772487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.772479    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2254cf9a-58ee-46db-86dd-e17faaee6caf-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:01.772487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:01.772494    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvb7p\" (UniqueName: \"kubernetes.io/projected/2254cf9a-58ee-46db-86dd-e17faaee6caf-kube-api-access-hvb7p\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:02.118458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.118422    5042 generic.go:356] "Generic (PLEG): container finished" podID="2254cf9a-58ee-46db-86dd-e17faaee6caf" containerID="7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531" exitCode=0
Nov 02 18:33:02.118667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.118476    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"
Nov 02 18:33:02.118667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.118514    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw" event={"ID":"2254cf9a-58ee-46db-86dd-e17faaee6caf","Type":"ContainerDied","Data":"7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531"}
Nov 02 18:33:02.118667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.118578    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw" event={"ID":"2254cf9a-58ee-46db-86dd-e17faaee6caf","Type":"ContainerDied","Data":"b2069988705bb10f4dc97594a14fb316a0737a820ed41214a4438098b04fed9b"}
Nov 02 18:33:02.118667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.118598    5042 scope.go:122] "RemoveContainer" containerID="7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531"
Nov 02 18:33:02.120317 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.120294    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp" event={"ID":"e521b386-8648-4cd7-94e6-e1ef84a7ac84","Type":"ContainerDied","Data":"a7e6a4f6b73e57db66f86d9327e9c3ec09a3900c3e3fe1fa8808980da27d2922"}
Nov 02 18:33:02.120412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.120344    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7e6a4f6b73e57db66f86d9327e9c3ec09a3900c3e3fe1fa8808980da27d2922"
Nov 02 18:33:02.120412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.120344    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"
Nov 02 18:33:02.130299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.130284    5042 scope.go:122] "RemoveContainer" containerID="7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531"
Nov 02 18:33:02.130578 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:02.130562    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531\": container with ID starting with 7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531 not found: ID does not exist" containerID="7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531"
Nov 02 18:33:02.130629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.130586    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531"} err="failed to get container status \"7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531\": rpc error: code = NotFound desc = could not find container \"7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531\": container with ID starting with 7af8f8c106c2da58754f09b6bce7dc6a9797f00f41e188afa0d8865f5e630531 not found: ID does not exist"
Nov 02 18:33:02.145066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.145035    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"]
Nov 02 18:33:02.151505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:02.151471    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2db162-86b845b866-rwwpw"]
Nov 02 18:33:03.542759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:03.542725    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2254cf9a-58ee-46db-86dd-e17faaee6caf" path="/var/lib/kubelet/pods/2254cf9a-58ee-46db-86dd-e17faaee6caf/volumes"
Nov 02 18:33:08.109628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.109566    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"]
Nov 02 18:33:08.110358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.110326    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e521b386-8648-4cd7-94e6-e1ef84a7ac84" containerName="revoke-keys"
Nov 02 18:33:08.110358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.110356    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e521b386-8648-4cd7-94e6-e1ef84a7ac84" containerName="revoke-keys"
Nov 02 18:33:08.110525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.110384    5042 cpu_manager.go:414] "containerMap: removing container" podUID="2254cf9a-58ee-46db-86dd-e17faaee6caf" containerName="maas-api"
Nov 02 18:33:08.110525 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.110394    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2254cf9a-58ee-46db-86dd-e17faaee6caf" containerName="maas-api"
Nov 02 18:33:08.110614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.110535    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e521b386-8648-4cd7-94e6-e1ef84a7ac84" containerName="revoke-keys"
Nov 02 18:33:08.110614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.110551    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="2254cf9a-58ee-46db-86dd-e17faaee6caf" containerName="maas-api"
Nov 02 18:33:08.124893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.116432    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.124893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.124293    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"]
Nov 02 18:33:08.124893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.124450    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:33:08.129777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.128284    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:08.129777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.128374    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:33:08.129777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.128516    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:33:08.151605 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.148128    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85hcg\" (UniqueName: \"kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.151605 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.148225    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.151605 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.148390    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.280867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.280790    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85hcg\" (UniqueName: \"kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.281097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.280905    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.281097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.280993    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.287662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.284634    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.291550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.291466    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.319792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.319757    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85hcg\" (UniqueName: \"kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg\") pod \"maas-api-revoke-sub-3375b230-sf592\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.459061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.459012    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:08.754376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:08.754348    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"]
Nov 02 18:33:08.755658 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:33:08.755633    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9de7f4_d729_4a03_8456_113fe6d8e55a.slice/crio-12acd039e4f0c190bf63774ad75fc0b642a1d93138b3b154a579508722d9a843 WatchSource:0}: Error finding container 12acd039e4f0c190bf63774ad75fc0b642a1d93138b3b154a579508722d9a843: Status 404 returned error can't find the container with id 12acd039e4f0c190bf63774ad75fc0b642a1d93138b3b154a579508722d9a843
Nov 02 18:33:09.174823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.174774    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592" event={"ID":"4f9de7f4-d729-4a03-8456-113fe6d8e55a","Type":"ContainerStarted","Data":"0c69f0b7ce0775e62772e9e50b1c24ffc9ba80e04884c92f0c8c5ee70961274c"}
Nov 02 18:33:09.175368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.174906    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592" event={"ID":"4f9de7f4-d729-4a03-8456-113fe6d8e55a","Type":"ContainerStarted","Data":"12acd039e4f0c190bf63774ad75fc0b642a1d93138b3b154a579508722d9a843"}
Nov 02 18:33:09.193040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.192982    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592" podStartSLOduration=1.192968247 podStartE2EDuration="1.192968247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:33:09.191884457 +0000 UTC m=+3601.771293842" watchObservedRunningTime="2026-11-02 18:33:09.192968247 +0000 UTC m=+3601.772377689"
Nov 02 18:33:09.848560 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.848527    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:33:09.851688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.851619    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:33:09.854833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.854800    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:33:09.857756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:09.857731    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:33:10.184008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:10.183971    5042 generic.go:356] "Generic (PLEG): container finished" podID="4f9de7f4-d729-4a03-8456-113fe6d8e55a" containerID="0c69f0b7ce0775e62772e9e50b1c24ffc9ba80e04884c92f0c8c5ee70961274c" exitCode=0
Nov 02 18:33:10.184398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:10.184016    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592" event={"ID":"4f9de7f4-d729-4a03-8456-113fe6d8e55a","Type":"ContainerDied","Data":"0c69f0b7ce0775e62772e9e50b1c24ffc9ba80e04884c92f0c8c5ee70961274c"}
Nov 02 18:33:11.289026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.280276    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"]
Nov 02 18:33:11.289026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.285077    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.310118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.310083    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-9165c3-openshift-default-dockercfg-7wchl\""
Nov 02 18:33:11.316211 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.316144    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"]
Nov 02 18:33:11.328642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.327312    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.331993 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.331943    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlcb\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332080    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332151    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332271    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332460 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332432    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332541    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332609    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.332712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.332699    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.395316 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.395281    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:11.433806 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.433771    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca\") pod \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") "
Nov 02 18:33:11.433806 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.433819    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca\") pod \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") "
Nov 02 18:33:11.434063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.433941    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg\" (UniqueName: \"kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg\") pod \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\" (UID: \"4f9de7f4-d729-4a03-8456-113fe6d8e55a\") "
Nov 02 18:33:11.434221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434194    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434254    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca" pod "4f9de7f4-d729-4a03-8456-113fe6d8e55a" (UID: "4f9de7f4-d729-4a03-8456-113fe6d8e55a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:11.434336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434311    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434342    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlcb\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434390    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434433    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434470    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434561    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434625    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434662    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.434681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434676    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.435079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434687    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.435079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.434778    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:11.435209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.435173    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.435434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.435405    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.435715 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.435692    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.437232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.437200    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca" pod "4f9de7f4-d729-4a03-8456-113fe6d8e55a" (UID: "4f9de7f4-d729-4a03-8456-113fe6d8e55a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:11.437724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.437688    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.437869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.437730    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg" pod "4f9de7f4-d729-4a03-8456-113fe6d8e55a" (UID: "4f9de7f4-d729-4a03-8456-113fe6d8e55a"). InnerVolumeSpecName "kube-api-access-85hcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:11.438040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.438020    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.442903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.442773    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlcb\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.443356 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.443333    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token\") pod \"e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") " pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.536385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.536336    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9de7f4-d729-4a03-8456-113fe6d8e55a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:11.536385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.536371    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85hcg\" (UniqueName: \"kubernetes.io/projected/4f9de7f4-d729-4a03-8456-113fe6d8e55a-kube-api-access-85hcg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:11.625087 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.625044    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:11.937785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.937757    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"]
Nov 02 18:33:11.938568 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:33:11.938542    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595396f2_d32e_4547_89cd_15d5e10c17dc.slice/crio-7b3afdd84cf075652a135897af2d98dda6d85041a3732f86b001fd1ae5856f10 WatchSource:0}: Error finding container 7b3afdd84cf075652a135897af2d98dda6d85041a3732f86b001fd1ae5856f10: Status 404 returned error can't find the container with id 7b3afdd84cf075652a135897af2d98dda6d85041a3732f86b001fd1ae5856f10
Nov 02 18:33:11.941119 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.941083    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:33:11.941237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.941165    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:33:11.941237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:11.941208    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:33:12.205563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.205516    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" event={"ID":"595396f2-d32e-4547-89cd-15d5e10c17dc","Type":"ContainerStarted","Data":"f717be38bc6089a9e3c5978a8b57e8149238dcf8100becf9b8ecfda12f1d03b4"}
Nov 02 18:33:12.206033 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.205957    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" event={"ID":"595396f2-d32e-4547-89cd-15d5e10c17dc","Type":"ContainerStarted","Data":"7b3afdd84cf075652a135897af2d98dda6d85041a3732f86b001fd1ae5856f10"}
Nov 02 18:33:12.207691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.207665    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"
Nov 02 18:33:12.207873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.207758    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592" event={"ID":"4f9de7f4-d729-4a03-8456-113fe6d8e55a","Type":"ContainerDied","Data":"12acd039e4f0c190bf63774ad75fc0b642a1d93138b3b154a579508722d9a843"}
Nov 02 18:33:12.207873 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.207799    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12acd039e4f0c190bf63774ad75fc0b642a1d93138b3b154a579508722d9a843"
Nov 02 18:33:12.225230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.225162    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podStartSLOduration=1.225140258 podStartE2EDuration="1.225140258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:33:12.223559381 +0000 UTC m=+3604.802968768" watchObservedRunningTime="2026-11-02 18:33:12.225140258 +0000 UTC m=+3604.804549656"
Nov 02 18:33:12.626295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:12.626190    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:13.627254 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:13.627217    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 18:33:13.627713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:13.627293    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 18:33:14.627506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:14.627470    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 18:33:14.628206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:14.628171    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 18:33:15.625561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:15.625516    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 18:33:15.625756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:15.625598    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 18:33:16.626205 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:16.626157    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 18:33:16.626758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:16.626243    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 18:33:17.628897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:17.627304    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 18:33:17.628897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:17.627372    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 18:33:18.545736 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:18.545669    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:33:18.676995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.676956    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:18.677590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.677479    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:18.678550 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.678522    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:33:18.750134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.750096    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"]
Nov 02 18:33:18.751322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.751293    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4f9de7f4-d729-4a03-8456-113fe6d8e55a" containerName="revoke-keys"
Nov 02 18:33:18.751502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.751488    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9de7f4-d729-4a03-8456-113fe6d8e55a" containerName="revoke-keys"
Nov 02 18:33:18.751724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.751710    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9de7f4-d729-4a03-8456-113fe6d8e55a" containerName="revoke-keys"
Nov 02 18:33:18.756098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.756073    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"]
Nov 02 18:33:18.756386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.756332    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.762109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.761088    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:18.762109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.761952    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:33:18.762109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.762026    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:33:18.762109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.762079    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:33:18.885031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.884987    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vssq\" (UniqueName: \"kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.885265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.885208    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.885628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.885603    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.987476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.987437    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.987695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.987532    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.987695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.987663    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vssq\" (UniqueName: \"kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:18.988756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:18.988718    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:19.010910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:19.010860    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:19.018914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:19.017228    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vssq\" (UniqueName: \"kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq\") pod \"maas-api-revoke-sub-cee3a495-tfkl6\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:19.106562 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:19.106478    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:19.393038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:19.393005    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"]
Nov 02 18:33:19.393868 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:33:19.393810    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6d802e_f256_4e44_801e_dc891654530c.slice/crio-bd2f54c63f4db695855674345cfa5b07f348b67050358d60fe3eb24045d6ef6f WatchSource:0}: Error finding container bd2f54c63f4db695855674345cfa5b07f348b67050358d60fe3eb24045d6ef6f: Status 404 returned error can't find the container with id bd2f54c63f4db695855674345cfa5b07f348b67050358d60fe3eb24045d6ef6f
Nov 02 18:33:20.292576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:20.292530    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6" event={"ID":"bf6d802e-f256-4e44-801e-dc891654530c","Type":"ContainerStarted","Data":"8b4089685a62a160efb785785d040030600ec9da97d8e0a80a5454024bcd1da7"}
Nov 02 18:33:20.293166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:20.292604    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6" event={"ID":"bf6d802e-f256-4e44-801e-dc891654530c","Type":"ContainerStarted","Data":"bd2f54c63f4db695855674345cfa5b07f348b67050358d60fe3eb24045d6ef6f"}
Nov 02 18:33:20.311981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:20.311908    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6" podStartSLOduration=2.3118871260000002 podStartE2EDuration="2.311887126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:33:20.307614769 +0000 UTC m=+3612.887024158" watchObservedRunningTime="2026-11-02 18:33:20.311887126 +0000 UTC m=+3612.891296513"
Nov 02 18:33:21.300762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:21.300721    5042 generic.go:356] "Generic (PLEG): container finished" podID="bf6d802e-f256-4e44-801e-dc891654530c" containerID="8b4089685a62a160efb785785d040030600ec9da97d8e0a80a5454024bcd1da7" exitCode=0
Nov 02 18:33:21.301259 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:21.300820    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6" event={"ID":"bf6d802e-f256-4e44-801e-dc891654530c","Type":"ContainerDied","Data":"8b4089685a62a160efb785785d040030600ec9da97d8e0a80a5454024bcd1da7"}
Nov 02 18:33:22.406235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.399490    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"]
Nov 02 18:33:22.406235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.399697    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" containerID="cri-o://1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726" gracePeriod=30
Nov 02 18:33:22.632683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.632653    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:22.672180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.672135    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca\") pod \"bf6d802e-f256-4e44-801e-dc891654530c\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") "
Nov 02 18:33:22.672387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.672258    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq\" (UniqueName: \"kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq\") pod \"bf6d802e-f256-4e44-801e-dc891654530c\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") "
Nov 02 18:33:22.672459 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.672430    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca\") pod \"bf6d802e-f256-4e44-801e-dc891654530c\" (UID: \"bf6d802e-f256-4e44-801e-dc891654530c\") "
Nov 02 18:33:22.673298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.673264    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca" pod "bf6d802e-f256-4e44-801e-dc891654530c" (UID: "bf6d802e-f256-4e44-801e-dc891654530c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:22.676519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.676478    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca" pod "bf6d802e-f256-4e44-801e-dc891654530c" (UID: "bf6d802e-f256-4e44-801e-dc891654530c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:22.676760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.676735    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq" pod "bf6d802e-f256-4e44-801e-dc891654530c" (UID: "bf6d802e-f256-4e44-801e-dc891654530c"). InnerVolumeSpecName "kube-api-access-9vssq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:22.775631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.775575    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vssq\" (UniqueName: \"kubernetes.io/projected/bf6d802e-f256-4e44-801e-dc891654530c-kube-api-access-9vssq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:22.775631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.775616    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:22.775631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:22.775633    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6d802e-f256-4e44-801e-dc891654530c-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:23.319652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:23.319611    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6" event={"ID":"bf6d802e-f256-4e44-801e-dc891654530c","Type":"ContainerDied","Data":"bd2f54c63f4db695855674345cfa5b07f348b67050358d60fe3eb24045d6ef6f"}
Nov 02 18:33:23.319829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:23.319690    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd2f54c63f4db695855674345cfa5b07f348b67050358d60fe3eb24045d6ef6f"
Nov 02 18:33:23.319829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:23.319624    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"
Nov 02 18:33:25.248275 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:25.248235    5042 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:33:25.248670 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:25.248315    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:33:27.693188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.693152    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 18:33:27.760422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760385    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760477    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760518    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760544    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760605    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760642 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760639    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760670    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.760948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760889    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:27.760948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760899    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.760910    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761010    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761019    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo\") pod \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\" (UID: \"a7f975ec-36ec-401d-b5a8-cca7fcc299fb\") "
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761044    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761573    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761595    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761609    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761623    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.762044 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.761816    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:27.763320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.763296    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:33:27.763441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.763350    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:27.763505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.763452    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:27.763595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.763574    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n" pod "a7f975ec-36ec-401d-b5a8-cca7fcc299fb" (UID: "a7f975ec-36ec-401d-b5a8-cca7fcc299fb"). InnerVolumeSpecName "kube-api-access-srh6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:27.862823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.862778    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srh6n\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-kube-api-access-srh6n\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.862823 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.862822    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.863067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.862867    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.863067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.862877    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:27.863067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:27.862886    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f975ec-36ec-401d-b5a8-cca7fcc299fb-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:28.379437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.379400    5042 generic.go:356] "Generic (PLEG): container finished" podID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerID="1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726" exitCode=0
Nov 02 18:33:28.379620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.379465    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"
Nov 02 18:33:28.379620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.379492    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" event={"ID":"a7f975ec-36ec-401d-b5a8-cca7fcc299fb","Type":"ContainerDied","Data":"1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726"}
Nov 02 18:33:28.379620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.379570    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql" event={"ID":"a7f975ec-36ec-401d-b5a8-cca7fcc299fb","Type":"ContainerDied","Data":"953d06aac9a7902ebfbd11d87014ccbbf4df47ac70cc6de8f9f56de188cf7837"}
Nov 02 18:33:28.379620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.379593    5042 scope.go:122] "RemoveContainer" containerID="1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726"
Nov 02 18:33:28.392386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.392366    5042 scope.go:122] "RemoveContainer" containerID="3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"
Nov 02 18:33:28.404272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.404249    5042 scope.go:122] "RemoveContainer" containerID="1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726"
Nov 02 18:33:28.404657 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:28.404630    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726\": container with ID starting with 1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726 not found: ID does not exist" containerID="1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726"
Nov 02 18:33:28.404718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.404658    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726"} err="failed to get container status \"1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726\": rpc error: code = NotFound desc = could not find container \"1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726\": container with ID starting with 1ab86ff09342553f9813d550b1f41df5a9f2432f012b6e881714b486ee593726 not found: ID does not exist"
Nov 02 18:33:28.404718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.404676    5042 scope.go:122] "RemoveContainer" containerID="3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"
Nov 02 18:33:28.404988 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:28.404969    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb\": container with ID starting with 3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb not found: ID does not exist" containerID="3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"
Nov 02 18:33:28.405057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.404994    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb"} err="failed to get container status \"3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb\": rpc error: code = NotFound desc = could not find container \"3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb\": container with ID starting with 3a29f35a55540f901be674347187f5179ba347262852ba51c20115a4ffb7defb not found: ID does not exist"
Nov 02 18:33:28.406635 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.406611    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"]
Nov 02 18:33:28.411829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:28.411804    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1e2281-openshift-default-5b4778f47c-2qhql"]
Nov 02 18:33:29.108639 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.108600    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"]
Nov 02 18:33:29.109132 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109118    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy"
Nov 02 18:33:29.109182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109133    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy"
Nov 02 18:33:29.109182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109158    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bf6d802e-f256-4e44-801e-dc891654530c" containerName="revoke-keys"
Nov 02 18:33:29.109182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109163    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6d802e-f256-4e44-801e-dc891654530c" containerName="revoke-keys"
Nov 02 18:33:29.109292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109255    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6d802e-f256-4e44-801e-dc891654530c" containerName="revoke-keys"
Nov 02 18:33:29.109292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109266    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy"
Nov 02 18:33:29.109292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.109275    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy"
Nov 02 18:33:29.114194 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.114172    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.122076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.122045    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:33:29.122076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.122048    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:33:29.122076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.122061    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:29.122374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.122134    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:33:29.125308 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.125281    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"]
Nov 02 18:33:29.182221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.182179    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.182427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.182329    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhc9\" (UniqueName: \"kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.182487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.182426    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.284055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.284010    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.284258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.284115    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhc9\" (UniqueName: \"kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.284258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.284161    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.284830 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.284761    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.286696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.286663    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.292688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.292663    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhc9\" (UniqueName: \"kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9\") pod \"maas-api-revoke-sub-955448eb-6lc8l\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.431627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.431523    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:29.549152 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.549115    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" path="/var/lib/kubelet/pods/a7f975ec-36ec-401d-b5a8-cca7fcc299fb/volumes"
Nov 02 18:33:29.739107 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:29.739077    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"]
Nov 02 18:33:29.741762 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:33:29.741727    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39499861_cad8_4bfa_a126_8c16f0158d16.slice/crio-d33c4d17cc5e20d4b225b4435f5f7ec4e133393c2013079e4c6e59cb118a8a2f WatchSource:0}: Error finding container d33c4d17cc5e20d4b225b4435f5f7ec4e133393c2013079e4c6e59cb118a8a2f: Status 404 returned error can't find the container with id d33c4d17cc5e20d4b225b4435f5f7ec4e133393c2013079e4c6e59cb118a8a2f
Nov 02 18:33:30.399386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:30.399289    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l" event={"ID":"39499861-cad8-4bfa-a126-8c16f0158d16","Type":"ContainerStarted","Data":"062f6a4aab7c0d81b17409003016f6805a3bc43e1b54547af53715e393e93618"}
Nov 02 18:33:30.399386 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:30.399343    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l" event={"ID":"39499861-cad8-4bfa-a126-8c16f0158d16","Type":"ContainerStarted","Data":"d33c4d17cc5e20d4b225b4435f5f7ec4e133393c2013079e4c6e59cb118a8a2f"}
Nov 02 18:33:30.415412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:30.415345    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l" podStartSLOduration=1.415325068 podStartE2EDuration="1.415325068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:33:30.414401753 +0000 UTC m=+3622.993811140" watchObservedRunningTime="2026-11-02 18:33:30.415325068 +0000 UTC m=+3622.994734455"
Nov 02 18:33:31.411067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:31.411032    5042 generic.go:356] "Generic (PLEG): container finished" podID="39499861-cad8-4bfa-a126-8c16f0158d16" containerID="062f6a4aab7c0d81b17409003016f6805a3bc43e1b54547af53715e393e93618" exitCode=0
Nov 02 18:33:31.411656 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:31.411117    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l" event={"ID":"39499861-cad8-4bfa-a126-8c16f0158d16","Type":"ContainerDied","Data":"062f6a4aab7c0d81b17409003016f6805a3bc43e1b54547af53715e393e93618"}
Nov 02 18:33:32.588363 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.588331    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:32.739068 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.739004    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca\") pod \"39499861-cad8-4bfa-a126-8c16f0158d16\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") "
Nov 02 18:33:32.739269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.739198    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca\") pod \"39499861-cad8-4bfa-a126-8c16f0158d16\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") "
Nov 02 18:33:32.739269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.739218    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9\" (UniqueName: \"kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9\") pod \"39499861-cad8-4bfa-a126-8c16f0158d16\" (UID: \"39499861-cad8-4bfa-a126-8c16f0158d16\") "
Nov 02 18:33:32.739407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.739381    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca" pod "39499861-cad8-4bfa-a126-8c16f0158d16" (UID: "39499861-cad8-4bfa-a126-8c16f0158d16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:32.739644 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.739620    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:32.741909 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.741821    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9" pod "39499861-cad8-4bfa-a126-8c16f0158d16" (UID: "39499861-cad8-4bfa-a126-8c16f0158d16"). InnerVolumeSpecName "kube-api-access-shhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:32.742058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.741907    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca" pod "39499861-cad8-4bfa-a126-8c16f0158d16" (UID: "39499861-cad8-4bfa-a126-8c16f0158d16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:32.841191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.841158    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39499861-cad8-4bfa-a126-8c16f0158d16-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:32.841191 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:32.841188    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhc9\" (UniqueName: \"kubernetes.io/projected/39499861-cad8-4bfa-a126-8c16f0158d16-kube-api-access-shhc9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:33.262504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:33.262463    5042 scope.go:122] "RemoveContainer" containerID="c84c0579baaa84d607b506475a9749175a08c7c69aec2825d2d5a7c1315ced2d"
Nov 02 18:33:33.441958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:33.441916    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"
Nov 02 18:33:33.441958 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:33.441940    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l" event={"ID":"39499861-cad8-4bfa-a126-8c16f0158d16","Type":"ContainerDied","Data":"d33c4d17cc5e20d4b225b4435f5f7ec4e133393c2013079e4c6e59cb118a8a2f"}
Nov 02 18:33:33.442288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:33.442017    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33c4d17cc5e20d4b225b4435f5f7ec4e133393c2013079e4c6e59cb118a8a2f"
Nov 02 18:33:37.885527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:37.885367    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"]
Nov 02 18:33:37.886066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:37.885696    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" containerID="cri-o://49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f" gracePeriod=30
Nov 02 18:33:38.733952 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.733909    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"]
Nov 02 18:33:38.734420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.734407    5042 cpu_manager.go:414] "containerMap: removing container" podUID="39499861-cad8-4bfa-a126-8c16f0158d16" containerName="revoke-keys"
Nov 02 18:33:38.734477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.734421    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39499861-cad8-4bfa-a126-8c16f0158d16" containerName="revoke-keys"
Nov 02 18:33:38.734477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.734439    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy"
Nov 02 18:33:38.734477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.734445    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f975ec-36ec-401d-b5a8-cca7fcc299fb" containerName="istio-proxy"
Nov 02 18:33:38.734588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.734560    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="39499861-cad8-4bfa-a126-8c16f0158d16" containerName="revoke-keys"
Nov 02 18:33:38.739630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.739609    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.751863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.746615    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"]
Nov 02 18:33:38.751863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.747730    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:38.751863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.748555    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:33:38.751863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.750801    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:33:38.751863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.750881    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:33:38.817946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.817895    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.818171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.817954    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.818241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.818176    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwhq\" (UniqueName: \"kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.919247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.919204    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.919730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.919277    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.919730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.919405    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwhq\" (UniqueName: \"kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.920090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.920062    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.922100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.922076    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:38.927673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:38.927648    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwhq\" (UniqueName: \"kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq\") pod \"maas-api-revoke-sub-76445f74-tx7zp\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:39.067697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:39.067599    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:39.354750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:39.354715    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"]
Nov 02 18:33:39.355901 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:33:39.355873    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e3fd54_5771_4d8a_a62c_149f4d517300.slice/crio-2a01933e177ed67e7952519e93fafdce3b800147e4951d77d863d24e6b7de8f0 WatchSource:0}: Error finding container 2a01933e177ed67e7952519e93fafdce3b800147e4951d77d863d24e6b7de8f0: Status 404 returned error can't find the container with id 2a01933e177ed67e7952519e93fafdce3b800147e4951d77d863d24e6b7de8f0
Nov 02 18:33:39.445161 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:39.445124    5042 patch_prober.go:29] interesting pod/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:33:39.445335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:39.445197    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:33:39.490188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:39.490144    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp" event={"ID":"a3e3fd54-5771-4d8a-a62c-149f4d517300","Type":"ContainerStarted","Data":"2a01933e177ed67e7952519e93fafdce3b800147e4951d77d863d24e6b7de8f0"}
Nov 02 18:33:40.500284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:40.500248    5042 generic.go:356] "Generic (PLEG): container finished" podID="a3e3fd54-5771-4d8a-a62c-149f4d517300" containerID="54bf3da646ac2e447c7c6dde0419b43fafe82c207b7745fdc685107f623496c0" exitCode=0
Nov 02 18:33:40.500705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:40.500329    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp" event={"ID":"a3e3fd54-5771-4d8a-a62c-149f4d517300","Type":"ContainerDied","Data":"54bf3da646ac2e447c7c6dde0419b43fafe82c207b7745fdc685107f623496c0"}
Nov 02 18:33:41.688930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.688902    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:41.861522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.861428    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca\") pod \"a3e3fd54-5771-4d8a-a62c-149f4d517300\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") "
Nov 02 18:33:41.861701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.861545    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca\") pod \"a3e3fd54-5771-4d8a-a62c-149f4d517300\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") "
Nov 02 18:33:41.861750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.861707    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq\" (UniqueName: \"kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq\") pod \"a3e3fd54-5771-4d8a-a62c-149f4d517300\" (UID: \"a3e3fd54-5771-4d8a-a62c-149f4d517300\") "
Nov 02 18:33:41.861892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.861865    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca" pod "a3e3fd54-5771-4d8a-a62c-149f4d517300" (UID: "a3e3fd54-5771-4d8a-a62c-149f4d517300"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:41.862199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.862173    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:41.864008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.863984    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca" pod "a3e3fd54-5771-4d8a-a62c-149f4d517300" (UID: "a3e3fd54-5771-4d8a-a62c-149f4d517300"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:33:41.864133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.864030    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq" pod "a3e3fd54-5771-4d8a-a62c-149f4d517300" (UID: "a3e3fd54-5771-4d8a-a62c-149f4d517300"). InnerVolumeSpecName "kube-api-access-cqwhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:41.963134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.963098    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e3fd54-5771-4d8a-a62c-149f4d517300-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:41.963134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:41.963130    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwhq\" (UniqueName: \"kubernetes.io/projected/a3e3fd54-5771-4d8a-a62c-149f4d517300-kube-api-access-cqwhq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:42.515324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:42.515288    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"
Nov 02 18:33:42.515517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:42.515330    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp" event={"ID":"a3e3fd54-5771-4d8a-a62c-149f4d517300","Type":"ContainerDied","Data":"2a01933e177ed67e7952519e93fafdce3b800147e4951d77d863d24e6b7de8f0"}
Nov 02 18:33:42.515517 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:42.515391    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a01933e177ed67e7952519e93fafdce3b800147e4951d77d863d24e6b7de8f0"
Nov 02 18:33:43.159176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.159108    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 18:33:43.279097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279061    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279190    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279232    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279271    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279328    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279358    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279415    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279442    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279458    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:43.279757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279513    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data\") pod \"4650ea1b-88c4-4333-b553-27f88d815e90\" (UID: \"4650ea1b-88c4-4333-b553-27f88d815e90\") "
Nov 02 18:33:43.279757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279608    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:43.279900 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279822    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:43.280093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.280069    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:43.280257 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.279995    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.280518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.280269    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.280518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.280287    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.280518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.280152    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:33:43.282092 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.282059    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:33:43.282270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.282249    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:43.282442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.282419    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "kube-api-access-xnk26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:33:43.283021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.283000    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo" pod "4650ea1b-88c4-4333-b553-27f88d815e90" (UID: "4650ea1b-88c4-4333-b553-27f88d815e90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:33:43.381518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.381477    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnk26\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-kube-api-access-xnk26\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.381518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.381512    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4650ea1b-88c4-4333-b553-27f88d815e90-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.381518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.381522    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.381786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.381533    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4650ea1b-88c4-4333-b553-27f88d815e90-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.381786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.381544    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4650ea1b-88c4-4333-b553-27f88d815e90-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.381786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.381552    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4650ea1b-88c4-4333-b553-27f88d815e90-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:33:43.523543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.523503    5042 generic.go:356] "Generic (PLEG): container finished" podID="4650ea1b-88c4-4333-b553-27f88d815e90" containerID="49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f" exitCode=0
Nov 02 18:33:43.523732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.523560    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"
Nov 02 18:33:43.523732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.523567    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" event={"ID":"4650ea1b-88c4-4333-b553-27f88d815e90","Type":"ContainerDied","Data":"49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f"}
Nov 02 18:33:43.523732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.523620    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2" event={"ID":"4650ea1b-88c4-4333-b553-27f88d815e90","Type":"ContainerDied","Data":"adf0d5601c154dac8a6ccdd2c377ef87d079c1c26298110dd3e733479b82b007"}
Nov 02 18:33:43.523732 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.523652    5042 scope.go:122] "RemoveContainer" containerID="49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f"
Nov 02 18:33:43.536407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.536376    5042 scope.go:122] "RemoveContainer" containerID="c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"
Nov 02 18:33:43.552063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.552034    5042 scope.go:122] "RemoveContainer" containerID="49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f"
Nov 02 18:33:43.552489 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:43.552459    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f\": container with ID starting with 49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f not found: ID does not exist" containerID="49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f"
Nov 02 18:33:43.552654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.552499    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f"} err="failed to get container status \"49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f\": rpc error: code = NotFound desc = could not find container \"49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f\": container with ID starting with 49a3bf36e0f0cd44bce6eee6a64bdfcc9aa9cc89fa672ccb62c8655d943af75f not found: ID does not exist"
Nov 02 18:33:43.552654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.552527    5042 scope.go:122] "RemoveContainer" containerID="c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"
Nov 02 18:33:43.553648 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:43.553620    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab\": container with ID starting with c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab not found: ID does not exist" containerID="c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"
Nov 02 18:33:43.553748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.553659    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab"} err="failed to get container status \"c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab\": rpc error: code = NotFound desc = could not find container \"c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab\": container with ID starting with c4855c16c2a2d8b57ad4dbc51e8df182d07b5cfd5a1bd9589ada5629ea0868ab not found: ID does not exist"
Nov 02 18:33:43.554474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.554458    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"]
Nov 02 18:33:43.564051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:43.564009    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-7771bf-openshift-default-b8d97467c-x7hb2"]
Nov 02 18:33:45.567866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:45.564789    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" path="/var/lib/kubelet/pods/4650ea1b-88c4-4333-b553-27f88d815e90/volumes"
Nov 02 18:33:48.539146 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:33:48.539065    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:33:55.116912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:55.116871    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"]
Nov 02 18:33:55.117449 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:55.117143    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy" containerID="cri-o://c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd" gracePeriod=30
Nov 02 18:33:59.736364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.736328    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"]
Nov 02 18:33:59.736921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.736905    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy"
Nov 02 18:33:59.737000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.736922    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy"
Nov 02 18:33:59.737000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.736945    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a3e3fd54-5771-4d8a-a62c-149f4d517300" containerName="revoke-keys"
Nov 02 18:33:59.737000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.736950    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e3fd54-5771-4d8a-a62c-149f4d517300" containerName="revoke-keys"
Nov 02 18:33:59.737165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.737056    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy"
Nov 02 18:33:59.737165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.737068    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e3fd54-5771-4d8a-a62c-149f4d517300" containerName="revoke-keys"
Nov 02 18:33:59.741894 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.741871    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.745804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.745779    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:33:59.747019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.746985    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:33:59.747366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.747345    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:33:59.747448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.747406    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:33:59.747730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.747710    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"]
Nov 02 18:33:59.798626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.798596    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz5r\" (UniqueName: \"kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.798816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.798659    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.798927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.798814    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.900209 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.900170    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz5r\" (UniqueName: \"kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.900440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.900257    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.900440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.900307    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.901015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.900991    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.903056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.903029    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:33:59.908716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:33:59.908687    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz5r\" (UniqueName: \"kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r\") pod \"maas-api-revoke-sub-1d6fcd93-fj4tf\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:34:00.060567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.060483    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:34:00.344040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.344007    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"]
Nov 02 18:34:00.344628 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:34:00.344578    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6adb9307_bac8_46a8_8ff8_fa45185feba5.slice/crio-22c4dfac9ee7b7a1fbeddc9fa9da65c27f62833bbab2d1f788fa09fb700d818c WatchSource:0}: Error finding container 22c4dfac9ee7b7a1fbeddc9fa9da65c27f62833bbab2d1f788fa09fb700d818c: Status 404 returned error can't find the container with id 22c4dfac9ee7b7a1fbeddc9fa9da65c27f62833bbab2d1f788fa09fb700d818c
Nov 02 18:34:00.416916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.416883    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:34:00.507349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507316    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507349 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507351    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507371    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507478    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507528    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507571    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507649    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507675    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507705    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy\") pod \"c74beb22-3816-4661-aa11-2c034967e021\" (UID: \"c74beb22-3816-4661-aa11-2c034967e021\") "
Nov 02 18:34:00.507810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507715    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:00.508071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.507908    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:00.508071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.508048    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:00.508304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.508273    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:00.508304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.508288    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:00.508431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.508370    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.508431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.508394    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.508431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.508409    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c74beb22-3816-4661-aa11-2c034967e021-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.510182 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.510161    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:00.510428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.510401    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:34:00.510552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.510533    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:00.510609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.510562    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf" pod "c74beb22-3816-4661-aa11-2c034967e021" (UID: "c74beb22-3816-4661-aa11-2c034967e021"). InnerVolumeSpecName "kube-api-access-5d8lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:00.610347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.610254    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c74beb22-3816-4661-aa11-2c034967e021-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.610347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.610288    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.610347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.610303    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.610347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.610316    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.610347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.610329    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c74beb22-3816-4661-aa11-2c034967e021-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.610347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.610343    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8lf\" (UniqueName: \"kubernetes.io/projected/c74beb22-3816-4661-aa11-2c034967e021-kube-api-access-5d8lf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:00.687791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.687756    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf" event={"ID":"6adb9307-bac8-46a8-8ff8-fa45185feba5","Type":"ContainerStarted","Data":"22c4dfac9ee7b7a1fbeddc9fa9da65c27f62833bbab2d1f788fa09fb700d818c"}
Nov 02 18:34:00.689710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.689683    5042 generic.go:356] "Generic (PLEG): container finished" podID="c74beb22-3816-4661-aa11-2c034967e021" containerID="c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd" exitCode=0
Nov 02 18:34:00.689887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.689743    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"
Nov 02 18:34:00.689887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.689765    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" event={"ID":"c74beb22-3816-4661-aa11-2c034967e021","Type":"ContainerDied","Data":"c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd"}
Nov 02 18:34:00.689887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.689830    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57" event={"ID":"c74beb22-3816-4661-aa11-2c034967e021","Type":"ContainerDied","Data":"6247b8d85d68b1199c676deeaff2c50118b08c3daf1815d9e41f4aa834fdcd1e"}
Nov 02 18:34:00.689887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.689875    5042 scope.go:122] "RemoveContainer" containerID="c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd"
Nov 02 18:34:00.701786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.701718    5042 scope.go:122] "RemoveContainer" containerID="830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"
Nov 02 18:34:00.712647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.712615    5042 scope.go:122] "RemoveContainer" containerID="c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd"
Nov 02 18:34:00.713409 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:00.713368    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd\": container with ID starting with c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd not found: ID does not exist" containerID="c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd"
Nov 02 18:34:00.713528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.713422    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd"} err="failed to get container status \"c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd\": rpc error: code = NotFound desc = could not find container \"c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd\": container with ID starting with c524d8a1ba199376cb0d84ed48c1df31216db9a3561bb3d531a3bc10465513fd not found: ID does not exist"
Nov 02 18:34:00.713528 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.713445    5042 scope.go:122] "RemoveContainer" containerID="830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"
Nov 02 18:34:00.713819 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:00.713797    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361\": container with ID starting with 830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361 not found: ID does not exist" containerID="830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"
Nov 02 18:34:00.713926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.713830    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361"} err="failed to get container status \"830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361\": rpc error: code = NotFound desc = could not find container \"830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361\": container with ID starting with 830fc40177f4564339f8f1ba7bb5af2c9644d95097a44f5b6af1601cebca7361 not found: ID does not exist"
Nov 02 18:34:00.717471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.717448    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"]
Nov 02 18:34:00.720616 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:00.720595    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5ec436-openshift-default-58bbc5478f-tlv57"]
Nov 02 18:34:01.549752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:01.549708    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c74beb22-3816-4661-aa11-2c034967e021" path="/var/lib/kubelet/pods/c74beb22-3816-4661-aa11-2c034967e021/volumes"
Nov 02 18:34:01.697855 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:01.697806    5042 generic.go:356] "Generic (PLEG): container finished" podID="6adb9307-bac8-46a8-8ff8-fa45185feba5" containerID="2834c53b27f9309a25f842963c0ec7e7120a4641069be51d12b1324645c0d500" exitCode=0
Nov 02 18:34:01.698065 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:01.697888    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf" event={"ID":"6adb9307-bac8-46a8-8ff8-fa45185feba5","Type":"ContainerDied","Data":"2834c53b27f9309a25f842963c0ec7e7120a4641069be51d12b1324645c0d500"}
Nov 02 18:34:02.931465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:02.931439    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:34:03.039024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.038932    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r\" (UniqueName: \"kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r\") pod \"6adb9307-bac8-46a8-8ff8-fa45185feba5\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") "
Nov 02 18:34:03.039024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.038978    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca\") pod \"6adb9307-bac8-46a8-8ff8-fa45185feba5\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") "
Nov 02 18:34:03.039024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.039002    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca\") pod \"6adb9307-bac8-46a8-8ff8-fa45185feba5\" (UID: \"6adb9307-bac8-46a8-8ff8-fa45185feba5\") "
Nov 02 18:34:03.039334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.039311    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca" pod "6adb9307-bac8-46a8-8ff8-fa45185feba5" (UID: "6adb9307-bac8-46a8-8ff8-fa45185feba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:03.039442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.039420    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:03.041354 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.041326    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca" pod "6adb9307-bac8-46a8-8ff8-fa45185feba5" (UID: "6adb9307-bac8-46a8-8ff8-fa45185feba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:03.041488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.041471    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r" pod "6adb9307-bac8-46a8-8ff8-fa45185feba5" (UID: "6adb9307-bac8-46a8-8ff8-fa45185feba5"). InnerVolumeSpecName "kube-api-access-gsz5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:03.140324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.140266    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsz5r\" (UniqueName: \"kubernetes.io/projected/6adb9307-bac8-46a8-8ff8-fa45185feba5-kube-api-access-gsz5r\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:03.140324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.140319    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6adb9307-bac8-46a8-8ff8-fa45185feba5-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:03.737413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.737377    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"
Nov 02 18:34:03.737597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.737412    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf" event={"ID":"6adb9307-bac8-46a8-8ff8-fa45185feba5","Type":"ContainerDied","Data":"22c4dfac9ee7b7a1fbeddc9fa9da65c27f62833bbab2d1f788fa09fb700d818c"}
Nov 02 18:34:03.737597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:03.737474    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22c4dfac9ee7b7a1fbeddc9fa9da65c27f62833bbab2d1f788fa09fb700d818c"
Nov 02 18:34:15.024697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:15.024649    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"]
Nov 02 18:34:15.028210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:15.028178    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cabc5d2a-z5585"]
Nov 02 18:34:15.545409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:15.545371    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278bdd67-726f-4833-9cc2-1134268339b5" path="/var/lib/kubelet/pods/278bdd67-726f-4833-9cc2-1134268339b5/volumes"
Nov 02 18:34:18.539800 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:18.539764    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:34:24.607781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:24.607734    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"]
Nov 02 18:34:24.608348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:24.607971    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy" containerID="cri-o://3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5" gracePeriod=30
Nov 02 18:34:26.022862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:26.022776    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"]
Nov 02 18:34:26.027111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:26.027075    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08029aa8-bnt2t"]
Nov 02 18:34:27.544141 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:27.544101    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3dafb96-b522-4fd8-861e-b30c41402b88" path="/var/lib/kubelet/pods/b3dafb96-b522-4fd8-861e-b30c41402b88/volumes"
Nov 02 18:34:29.027106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.027063    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"]
Nov 02 18:34:29.030404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.030373    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a9cee9-hvvqq"]
Nov 02 18:34:29.542456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.542421    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="201f4338-dc79-4142-b649-181ab2313ac8" path="/var/lib/kubelet/pods/201f4338-dc79-4142-b649-181ab2313ac8/volumes"
Nov 02 18:34:29.913494 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.912409    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 18:34:29.961936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.961891    5042 generic.go:356] "Generic (PLEG): container finished" podID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerID="3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5" exitCode=0
Nov 02 18:34:29.962137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.961948    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"
Nov 02 18:34:29.962137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.961968    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" event={"ID":"95fefb15-6fe5-42c9-b808-cc4fc34eae70","Type":"ContainerDied","Data":"3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5"}
Nov 02 18:34:29.962137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.962032    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq" event={"ID":"95fefb15-6fe5-42c9-b808-cc4fc34eae70","Type":"ContainerDied","Data":"4c4d80b39f32f4186791b57489611767de2e8044ca52fd0aa5f78f0f5e437402"}
Nov 02 18:34:29.962137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.962064    5042 scope.go:122] "RemoveContainer" containerID="3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5"
Nov 02 18:34:29.974610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.974581    5042 scope.go:122] "RemoveContainer" containerID="d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"
Nov 02 18:34:29.986884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.986821    5042 scope.go:122] "RemoveContainer" containerID="3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5"
Nov 02 18:34:29.987429 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:29.987403    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5\": container with ID starting with 3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5 not found: ID does not exist" containerID="3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5"
Nov 02 18:34:29.987584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.987437    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5"} err="failed to get container status \"3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5\": rpc error: code = NotFound desc = could not find container \"3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5\": container with ID starting with 3379fdbf7dee2aee52123c80503366c8b583615f9b3ede08dc9eca3a4cee46f5 not found: ID does not exist"
Nov 02 18:34:29.987584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.987456    5042 scope.go:122] "RemoveContainer" containerID="d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"
Nov 02 18:34:29.987811 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:29.987781    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8\": container with ID starting with d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8 not found: ID does not exist" containerID="d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"
Nov 02 18:34:29.987918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.987819    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8"} err="failed to get container status \"d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8\": rpc error: code = NotFound desc = could not find container \"d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8\": container with ID starting with d97b7dfbdc559a59eccac1308134d262b842b44929a49dea4ce00908d3b833f8 not found: ID does not exist"
Nov 02 18:34:29.989174 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989152    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989184    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989227    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989358    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989490 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989428    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989490 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989470    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989540    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989590    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989609    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket\") pod \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\" (UID: \"95fefb15-6fe5-42c9-b808-cc4fc34eae70\") "
Nov 02 18:34:29.989821 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989794    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:29.989946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.989892    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:29.990061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.990039    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:29.990136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.990118    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:29.990184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.990142    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:29.990184 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.990164    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:29.990981 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.990922    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:29.992994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.992815    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:29.993156 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.993002    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:29.993602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.993565    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "kube-api-access-n7lp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:29.998036 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:29.998005    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo" pod "95fefb15-6fe5-42c9-b808-cc4fc34eae70" (UID: "95fefb15-6fe5-42c9-b808-cc4fc34eae70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.090977    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.091020    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.091036    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.091049    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.091065    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.091078    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7lp6\" (UniqueName: \"kubernetes.io/projected/95fefb15-6fe5-42c9-b808-cc4fc34eae70-kube-api-access-n7lp6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.091084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.091092    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95fefb15-6fe5-42c9-b808-cc4fc34eae70-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:30.288610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.288569    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"]
Nov 02 18:34:30.291916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:30.291884    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-47e1c2-openshift-default-77968876dc-7tsmq"]
Nov 02 18:34:31.168120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:31.168080    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"]
Nov 02 18:34:31.543315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:31.543279    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" path="/var/lib/kubelet/pods/95fefb15-6fe5-42c9-b808-cc4fc34eae70/volumes"
Nov 02 18:34:33.321724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:33.321681    5042 scope.go:122] "RemoveContainer" containerID="5fe3d44e6a200e8916c6cdd056aedc00cea35059c031550eafd67143d63290a0"
Nov 02 18:34:36.169124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:36.169070    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing" containerID="cri-o://27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd" gracePeriod=25
Nov 02 18:34:36.752949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:36.752903    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:34:38.895745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:38.895697    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"]
Nov 02 18:34:38.896676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:38.896633    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy" containerID="cri-o://f717be38bc6089a9e3c5978a8b57e8149238dcf8100becf9b8ecfda12f1d03b4" gracePeriod=30
Nov 02 18:34:39.037961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.037919    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"]
Nov 02 18:34:39.040673 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.040642    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31bc1d13-ccm8c"]
Nov 02 18:34:39.543404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.543368    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3588267e-0d64-4fb2-be97-06b5776f28d4" path="/var/lib/kubelet/pods/3588267e-0d64-4fb2-be97-06b5776f28d4/volumes"
Nov 02 18:34:39.800382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800297    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"]
Nov 02 18:34:39.800788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800773    5042 cpu_manager.go:414] "containerMap: removing container" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy"
Nov 02 18:34:39.800788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800787    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800819    5042 cpu_manager.go:414] "containerMap: removing container" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800826    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800859    5042 cpu_manager.go:414] "containerMap: removing container" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800865    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800873    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800879    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800886    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800891    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800904    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6adb9307-bac8-46a8-8ff8-fa45185feba5" containerName="revoke-keys"
Nov 02 18:34:39.800966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800908    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6adb9307-bac8-46a8-8ff8-fa45185feba5" containerName="revoke-keys"
Nov 02 18:34:39.801431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800982    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6adb9307-bac8-46a8-8ff8-fa45185feba5" containerName="revoke-keys"
Nov 02 18:34:39.801431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.800992    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="4650ea1b-88c4-4333-b553-27f88d815e90" containerName="istio-proxy"
Nov 02 18:34:39.801431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.801002    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy"
Nov 02 18:34:39.801431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.801009    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="95fefb15-6fe5-42c9-b808-cc4fc34eae70" containerName="istio-proxy"
Nov 02 18:34:39.801431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.801015    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy"
Nov 02 18:34:39.801431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.801020    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c74beb22-3816-4661-aa11-2c034967e021" containerName="istio-proxy"
Nov 02 18:34:39.806270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.806248    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.810731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.810704    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:34:39.811072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.811055    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:34:39.812985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.812957    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:34:39.813092 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.812992    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:34:39.815741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.815719    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"]
Nov 02 18:34:39.837360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.837317    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.837587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.837444    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.837587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.837491    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5h6\" (UniqueName: \"kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.938212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.938175    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.951785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.938228    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5h6\" (UniqueName: \"kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.951785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.938335    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.951785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.938967    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.951785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.940874    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:39.952031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:39.951864    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5h6\" (UniqueName: \"kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6\") pod \"maas-api-revoke-sub-bfd8b867-pct8z\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:40.133180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:40.133097    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:40.424427 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:40.424394    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"]
Nov 02 18:34:40.425518 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:34:40.425481    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627c8adc_845f_431f_b9f5_695a18be41d9.slice/crio-d81b625f0b413f2b6deb751af9a231afd270eabb236f343c71ea9c232e7c5887 WatchSource:0}: Error finding container d81b625f0b413f2b6deb751af9a231afd270eabb236f343c71ea9c232e7c5887: Status 404 returned error can't find the container with id d81b625f0b413f2b6deb751af9a231afd270eabb236f343c71ea9c232e7c5887
Nov 02 18:34:41.045790 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:41.045747    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z" event={"ID":"627c8adc-845f-431f-b9f5-695a18be41d9","Type":"ContainerStarted","Data":"907d5a5b8c6ed4c1239e59bfee8f4e6db506a131f2680de0f4c07b196b6f0fed"}
Nov 02 18:34:41.046232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:41.045817    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z" event={"ID":"627c8adc-845f-431f-b9f5-695a18be41d9","Type":"ContainerStarted","Data":"d81b625f0b413f2b6deb751af9a231afd270eabb236f343c71ea9c232e7c5887"}
Nov 02 18:34:41.063575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:41.063522    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z" podStartSLOduration=2.06350935 podStartE2EDuration="2.06350935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:41.061758136 +0000 UTC m=+3693.641167522" watchObservedRunningTime="2026-11-02 18:34:41.06350935 +0000 UTC m=+3693.642918735"
Nov 02 18:34:42.053565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:42.053530    5042 generic.go:356] "Generic (PLEG): container finished" podID="627c8adc-845f-431f-b9f5-695a18be41d9" containerID="907d5a5b8c6ed4c1239e59bfee8f4e6db506a131f2680de0f4c07b196b6f0fed" exitCode=0
Nov 02 18:34:42.054026 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:42.053622    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z" event={"ID":"627c8adc-845f-431f-b9f5-695a18be41d9","Type":"ContainerDied","Data":"907d5a5b8c6ed4c1239e59bfee8f4e6db506a131f2680de0f4c07b196b6f0fed"}
Nov 02 18:34:43.218501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.218470    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:43.274269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.274171    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6\" (UniqueName: \"kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6\") pod \"627c8adc-845f-431f-b9f5-695a18be41d9\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") "
Nov 02 18:34:43.274269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.274260    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca\") pod \"627c8adc-845f-431f-b9f5-695a18be41d9\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") "
Nov 02 18:34:43.274551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.274466    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca\") pod \"627c8adc-845f-431f-b9f5-695a18be41d9\" (UID: \"627c8adc-845f-431f-b9f5-695a18be41d9\") "
Nov 02 18:34:43.274950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.274918    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca" pod "627c8adc-845f-431f-b9f5-695a18be41d9" (UID: "627c8adc-845f-431f-b9f5-695a18be41d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:43.276559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.276537    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca" pod "627c8adc-845f-431f-b9f5-695a18be41d9" (UID: "627c8adc-845f-431f-b9f5-695a18be41d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:43.276667 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.276629    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6" pod "627c8adc-845f-431f-b9f5-695a18be41d9" (UID: "627c8adc-845f-431f-b9f5-695a18be41d9"). InnerVolumeSpecName "kube-api-access-km5h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:43.375574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.375468    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5h6\" (UniqueName: \"kubernetes.io/projected/627c8adc-845f-431f-b9f5-695a18be41d9-kube-api-access-km5h6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:43.375574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.375502    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:43.375574 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:43.375517    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627c8adc-845f-431f-b9f5-695a18be41d9-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.068701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.068668    5042 generic.go:356] "Generic (PLEG): container finished" podID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerID="f717be38bc6089a9e3c5978a8b57e8149238dcf8100becf9b8ecfda12f1d03b4" exitCode=0
Nov 02 18:34:44.069039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.068711    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" event={"ID":"595396f2-d32e-4547-89cd-15d5e10c17dc","Type":"ContainerDied","Data":"f717be38bc6089a9e3c5978a8b57e8149238dcf8100becf9b8ecfda12f1d03b4"}
Nov 02 18:34:44.070891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.070555    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z" event={"ID":"627c8adc-845f-431f-b9f5-695a18be41d9","Type":"ContainerDied","Data":"d81b625f0b413f2b6deb751af9a231afd270eabb236f343c71ea9c232e7c5887"}
Nov 02 18:34:44.070891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.070613    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81b625f0b413f2b6deb751af9a231afd270eabb236f343c71ea9c232e7c5887"
Nov 02 18:34:44.070891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.070683    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"
Nov 02 18:34:44.141547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.141522    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:34:44.183433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183354    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183398    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183427    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183445    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183537    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183577    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183630    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183673    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.183746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183736    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token\") pod \"595396f2-d32e-4547-89cd-15d5e10c17dc\" (UID: \"595396f2-d32e-4547-89cd-15d5e10c17dc\") "
Nov 02 18:34:44.184077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183786    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:44.184077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.183829    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:44.184077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.184070    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:44.184249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.184120    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:44.184249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.184163    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.184249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.184200    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.184369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.184240    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:44.186690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.186662    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:44.187226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.187202    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "kube-api-access-9vlcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:44.187226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.187212    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:34:44.189087 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.189067    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy" pod "595396f2-d32e-4547-89cd-15d5e10c17dc" (UID: "595396f2-d32e-4547-89cd-15d5e10c17dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:44.285122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285082    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595396f2-d32e-4547-89cd-15d5e10c17dc-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.285122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285115    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.285122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285129    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.285602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285142    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595396f2-d32e-4547-89cd-15d5e10c17dc-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.285602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285155    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.285602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285167    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlcb\" (UniqueName: \"kubernetes.io/projected/595396f2-d32e-4547-89cd-15d5e10c17dc-kube-api-access-9vlcb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.285602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.285180    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595396f2-d32e-4547-89cd-15d5e10c17dc-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:44.478458 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.478418    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"]
Nov 02 18:34:44.490600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:44.490565    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"]
Nov 02 18:34:45.080299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:45.080268    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"
Nov 02 18:34:45.080299 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:45.080284    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm" event={"ID":"595396f2-d32e-4547-89cd-15d5e10c17dc","Type":"ContainerDied","Data":"7b3afdd84cf075652a135897af2d98dda6d85041a3732f86b001fd1ae5856f10"}
Nov 02 18:34:45.080603 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:45.080355    5042 scope.go:122] "RemoveContainer" containerID="f717be38bc6089a9e3c5978a8b57e8149238dcf8100becf9b8ecfda12f1d03b4"
Nov 02 18:34:45.111545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:45.111513    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"]
Nov 02 18:34:45.121171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:45.121144    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-9165c3-openshift-default-79758b8bf5-pcknm"]
Nov 02 18:34:45.543817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:45.543777    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" path="/var/lib/kubelet/pods/595396f2-d32e-4547-89cd-15d5e10c17dc/volumes"
Nov 02 18:34:46.752761 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:46.752719    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:34:48.540653 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:48.540617    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:34:51.184410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.184364    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"]
Nov 02 18:34:51.185166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.185140    5042 cpu_manager.go:414] "containerMap: removing container" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy"
Nov 02 18:34:51.185166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.185166    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy"
Nov 02 18:34:51.185373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.185197    5042 cpu_manager.go:414] "containerMap: removing container" podUID="627c8adc-845f-431f-b9f5-695a18be41d9" containerName="revoke-keys"
Nov 02 18:34:51.185373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.185206    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627c8adc-845f-431f-b9f5-695a18be41d9" containerName="revoke-keys"
Nov 02 18:34:51.185373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.185336    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="595396f2-d32e-4547-89cd-15d5e10c17dc" containerName="istio-proxy"
Nov 02 18:34:51.185373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.185356    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="627c8adc-845f-431f-b9f5-695a18be41d9" containerName="revoke-keys"
Nov 02 18:34:51.190669 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.190644    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"]
Nov 02 18:34:51.190793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.190769    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.194509 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.194464    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:34:51.195447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.195426    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:34:51.195600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.195434    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:34:51.195600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.195513    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:34:51.253054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.253016    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"]
Nov 02 18:34:51.253257 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.253233    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" podUID="78d07225-594c-4799-b7ca-6c7290cc9202" containerName="discovery" containerID="cri-o://5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1" gracePeriod=30
Nov 02 18:34:51.263527 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.263495    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.263692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.263613    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.263692 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.263654    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvjd\" (UniqueName: \"kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.364994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.364958    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.365192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.365035    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.365192 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.365080    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvjd\" (UniqueName: \"kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.365720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.365692    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.367548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.367529    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.374108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.374075    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvjd\" (UniqueName: \"kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd\") pod \"maas-api-revoke-sub-de953a5d-2xvr8\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.509874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.509822    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:51.531638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.531613    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 18:34:51.670455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670419    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.670662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670533    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.670662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670565    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.670662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670619    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.670801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670673    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.670801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670708    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.670801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.670740    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts\") pod \"78d07225-594c-4799-b7ca-6c7290cc9202\" (UID: \"78d07225-594c-4799-b7ca-6c7290cc9202\") "
Nov 02 18:34:51.671079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.671047    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:51.671261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.671240    5042 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-ca-configmap\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.674947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.674880    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "kube-api-access-q7gwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:51.675786 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.675754    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:51.675933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.675795    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:51.675933 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.675821    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:51.676060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.675943    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:51.676118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.676064    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts" pod "78d07225-594c-4799-b7ca-6c7290cc9202" (UID: "78d07225-594c-4799-b7ca-6c7290cc9202"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:51.772795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.772705    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.772795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.772736    5042 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-csr-dns-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.772795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.772746    5042 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-istio-kubeconfig\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.772795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.772756    5042 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d07225-594c-4799-b7ca-6c7290cc9202-local-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.772795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.772767    5042 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78d07225-594c-4799-b7ca-6c7290cc9202-cacerts\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.772795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.772777    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7gwb\" (UniqueName: \"kubernetes.io/projected/78d07225-594c-4799-b7ca-6c7290cc9202-kube-api-access-q7gwb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:51.805015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:51.804977    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"]
Nov 02 18:34:51.806392 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:34:51.806364    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e13bc1a_1c9d_47d1_8054_6e0cd1fd54b4.slice/crio-02871f59f1ace684fc19b1880caa7511ffc8a51829b1e6e80f75daf373c4308b WatchSource:0}: Error finding container 02871f59f1ace684fc19b1880caa7511ffc8a51829b1e6e80f75daf373c4308b: Status 404 returned error can't find the container with id 02871f59f1ace684fc19b1880caa7511ffc8a51829b1e6e80f75daf373c4308b
Nov 02 18:34:52.137796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.137758    5042 generic.go:356] "Generic (PLEG): container finished" podID="78d07225-594c-4799-b7ca-6c7290cc9202" containerID="5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1" exitCode=0
Nov 02 18:34:52.138002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.137810    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"
Nov 02 18:34:52.138002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.137868    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" event={"ID":"78d07225-594c-4799-b7ca-6c7290cc9202","Type":"ContainerDied","Data":"5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1"}
Nov 02 18:34:52.138002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.137920    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf" event={"ID":"78d07225-594c-4799-b7ca-6c7290cc9202","Type":"ContainerDied","Data":"ce8a00f6330472f4a4649dd57ce463ad7f6d0d10dd1bd488068fd9ebac15865c"}
Nov 02 18:34:52.138002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.137944    5042 scope.go:122] "RemoveContainer" containerID="5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1"
Nov 02 18:34:52.144876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.141328    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8" event={"ID":"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4","Type":"ContainerStarted","Data":"7ee5123d2603483927b95f95fa40f61c38351326b80d0da6038abfd4aa725464"}
Nov 02 18:34:52.144876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.141380    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8" event={"ID":"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4","Type":"ContainerStarted","Data":"02871f59f1ace684fc19b1880caa7511ffc8a51829b1e6e80f75daf373c4308b"}
Nov 02 18:34:52.153336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.153317    5042 scope.go:122] "RemoveContainer" containerID="5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1"
Nov 02 18:34:52.153704 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:52.153686    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1\": container with ID starting with 5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1 not found: ID does not exist" containerID="5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1"
Nov 02 18:34:52.153780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.153711    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1"} err="failed to get container status \"5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1\": rpc error: code = NotFound desc = could not find container \"5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1\": container with ID starting with 5a6aeba84d14fe47e68540699c31408e050e851d5684b52ea47bd5e39d1d15f1 not found: ID does not exist"
Nov 02 18:34:52.159892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.159817    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8" podStartSLOduration=1.159804942 podStartE2EDuration="1.159804942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:52.15712464 +0000 UTC m=+3704.736534029" watchObservedRunningTime="2026-11-02 18:34:52.159804942 +0000 UTC m=+3704.739214326"
Nov 02 18:34:52.171469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.171447    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"]
Nov 02 18:34:52.178290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:52.178264    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bktgf"]
Nov 02 18:34:53.153422 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:53.153381    5042 generic.go:356] "Generic (PLEG): container finished" podID="7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" containerID="7ee5123d2603483927b95f95fa40f61c38351326b80d0da6038abfd4aa725464" exitCode=0
Nov 02 18:34:53.154009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:53.153457    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8" event={"ID":"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4","Type":"ContainerDied","Data":"7ee5123d2603483927b95f95fa40f61c38351326b80d0da6038abfd4aa725464"}
Nov 02 18:34:53.543505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:53.543463    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d07225-594c-4799-b7ca-6c7290cc9202" path="/var/lib/kubelet/pods/78d07225-594c-4799-b7ca-6c7290cc9202/volumes"
Nov 02 18:34:54.326712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.326681    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:54.405061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.405018    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca\") pod \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") "
Nov 02 18:34:54.405226 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.405092    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca\") pod \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") "
Nov 02 18:34:54.405284 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.405271    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd\" (UniqueName: \"kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd\") pod \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\" (UID: \"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4\") "
Nov 02 18:34:54.405629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.405539    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca" pod "7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" (UID: "7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:54.405773 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.405756    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:54.407746 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.407721    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca" pod "7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" (UID: "7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:54.408251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.408233    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd" pod "7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" (UID: "7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4"). InnerVolumeSpecName "kube-api-access-2mvjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:54.507215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.507163    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvjd\" (UniqueName: \"kubernetes.io/projected/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-kube-api-access-2mvjd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:54.507215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:54.507199    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:55.167943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:55.167905    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8" event={"ID":"7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4","Type":"ContainerDied","Data":"02871f59f1ace684fc19b1880caa7511ffc8a51829b1e6e80f75daf373c4308b"}
Nov 02 18:34:55.167943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:55.167942    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"
Nov 02 18:34:55.168237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:55.167959    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02871f59f1ace684fc19b1880caa7511ffc8a51829b1e6e80f75daf373c4308b"
Nov 02 18:34:56.436424 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.436360    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 18:34:56.528144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.528101    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume\") pod \"a8c0ac11-411b-487c-afd9-abe76d467bba\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") "
Nov 02 18:34:56.528326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.528205    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6\" (UniqueName: \"kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6\") pod \"a8c0ac11-411b-487c-afd9-abe76d467bba\" (UID: \"a8c0ac11-411b-487c-afd9-abe76d467bba\") "
Nov 02 18:34:56.528505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.528482    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume" pod "a8c0ac11-411b-487c-afd9-abe76d467bba" (UID: "a8c0ac11-411b-487c-afd9-abe76d467bba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:34:56.528638 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.528624    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c0ac11-411b-487c-afd9-abe76d467bba-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:56.530515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.530493    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6" pod "a8c0ac11-411b-487c-afd9-abe76d467bba" (UID: "a8c0ac11-411b-487c-afd9-abe76d467bba"). InnerVolumeSpecName "kube-api-access-2pwn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:56.630431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:56.630375    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pwn6\" (UniqueName: \"kubernetes.io/projected/a8c0ac11-411b-487c-afd9-abe76d467bba-kube-api-access-2pwn6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:57.184137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.184097    5042 generic.go:356] "Generic (PLEG): container finished" podID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerID="27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd" exitCode=0
Nov 02 18:34:57.184379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.184171    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"
Nov 02 18:34:57.184379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.184199    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" event={"ID":"a8c0ac11-411b-487c-afd9-abe76d467bba","Type":"ContainerDied","Data":"27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd"}
Nov 02 18:34:57.184379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.184263    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9" event={"ID":"a8c0ac11-411b-487c-afd9-abe76d467bba","Type":"ContainerDied","Data":"1680e605a51b398da50b78f0a7241644cbca7f132e6df016ba80aa554639ad30"}
Nov 02 18:34:57.184379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.184287    5042 scope.go:122] "RemoveContainer" containerID="27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd"
Nov 02 18:34:57.198186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.198163    5042 scope.go:122] "RemoveContainer" containerID="27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd"
Nov 02 18:34:57.198500 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:34:57.198475    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd\": container with ID starting with 27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd not found: ID does not exist" containerID="27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd"
Nov 02 18:34:57.198604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.198510    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd"} err="failed to get container status \"27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd\": rpc error: code = NotFound desc = could not find container \"27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd\": container with ID starting with 27768cf488b445c99d3ad811f7257034728ebc1c11d8e5d3e09b6dad57fa56dd not found: ID does not exist"
Nov 02 18:34:57.209046 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.209006    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"]
Nov 02 18:34:57.212567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.212518    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-9165c3-7ccd478485-thjx9"]
Nov 02 18:34:57.545277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:57.545240    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" path="/var/lib/kubelet/pods/a8c0ac11-411b-487c-afd9-abe76d467bba/volumes"
Nov 02 18:34:59.501558 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.501509    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" containerID="cri-o://b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90" gracePeriod=45
Nov 02 18:34:59.518372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.518332    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" containerID="cri-o://c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3" gracePeriod=45
Nov 02 18:34:59.594617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.594575    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": dial tcp 10.132.0.66:8000: connect: connection refused"
Nov 02 18:34:59.785323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.785294    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:34:59.814718 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.814687    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:34:59.861676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.861639    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs\") pod \"831a619e-ccd4-40b5-9903-e83f3daca0c8\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") "
Nov 02 18:34:59.861924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.861714    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache\") pod \"831a619e-ccd4-40b5-9903-e83f3daca0c8\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") "
Nov 02 18:34:59.861924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.861833    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm\") pod \"831a619e-ccd4-40b5-9903-e83f3daca0c8\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") "
Nov 02 18:34:59.861924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.861912    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir\") pod \"831a619e-ccd4-40b5-9903-e83f3daca0c8\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") "
Nov 02 18:34:59.862210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.861962    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home\") pod \"831a619e-ccd4-40b5-9903-e83f3daca0c8\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") "
Nov 02 18:34:59.862210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.862008    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd\" (UniqueName: \"kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd\") pod \"831a619e-ccd4-40b5-9903-e83f3daca0c8\" (UID: \"831a619e-ccd4-40b5-9903-e83f3daca0c8\") "
Nov 02 18:34:59.863155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.863104    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir" pod "831a619e-ccd4-40b5-9903-e83f3daca0c8" (UID: "831a619e-ccd4-40b5-9903-e83f3daca0c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.863267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.863147    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home" pod "831a619e-ccd4-40b5-9903-e83f3daca0c8" (UID: "831a619e-ccd4-40b5-9903-e83f3daca0c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.863420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.863398    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache" pod "831a619e-ccd4-40b5-9903-e83f3daca0c8" (UID: "831a619e-ccd4-40b5-9903-e83f3daca0c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.865365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.865342    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs" pod "831a619e-ccd4-40b5-9903-e83f3daca0c8" (UID: "831a619e-ccd4-40b5-9903-e83f3daca0c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:59.865602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.865572    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd" pod "831a619e-ccd4-40b5-9903-e83f3daca0c8" (UID: "831a619e-ccd4-40b5-9903-e83f3daca0c8"). InnerVolumeSpecName "kube-api-access-gvhnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:34:59.865704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.865674    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm" pod "831a619e-ccd4-40b5-9903-e83f3daca0c8" (UID: "831a619e-ccd4-40b5-9903-e83f3daca0c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.963500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963462    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6\" (UniqueName: \"kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6\") pod \"329a6854-eaf7-4e3f-b080-ad6239cde483\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") "
Nov 02 18:34:59.963726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963512    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir\") pod \"329a6854-eaf7-4e3f-b080-ad6239cde483\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") "
Nov 02 18:34:59.963726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963532    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home\") pod \"329a6854-eaf7-4e3f-b080-ad6239cde483\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") "
Nov 02 18:34:59.963726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963566    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache\") pod \"329a6854-eaf7-4e3f-b080-ad6239cde483\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") "
Nov 02 18:34:59.963726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963668    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs\" (UniqueName: \"kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs\") pod \"329a6854-eaf7-4e3f-b080-ad6239cde483\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") "
Nov 02 18:34:59.963726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963683    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm\") pod \"329a6854-eaf7-4e3f-b080-ad6239cde483\" (UID: \"329a6854-eaf7-4e3f-b080-ad6239cde483\") "
Nov 02 18:34:59.964035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963870    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir" pod "329a6854-eaf7-4e3f-b080-ad6239cde483" (UID: "329a6854-eaf7-4e3f-b080-ad6239cde483"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.964035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963887    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home" pod "329a6854-eaf7-4e3f-b080-ad6239cde483" (UID: "329a6854-eaf7-4e3f-b080-ad6239cde483"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.964035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.963969    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache" pod "329a6854-eaf7-4e3f-b080-ad6239cde483" (UID: "329a6854-eaf7-4e3f-b080-ad6239cde483"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.964146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964070    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhnd\" (UniqueName: \"kubernetes.io/projected/831a619e-ccd4-40b5-9903-e83f3daca0c8-kube-api-access-gvhnd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964087    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964101    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964113    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/831a619e-ccd4-40b5-9903-e83f3daca0c8-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964126    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964139    5042 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-model-cache\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964153    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964165    5042 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-tmp-dir\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.964368 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.964176    5042 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/831a619e-ccd4-40b5-9903-e83f3daca0c8-home\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:34:59.966086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.966063    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm" pod "329a6854-eaf7-4e3f-b080-ad6239cde483" (UID: "329a6854-eaf7-4e3f-b080-ad6239cde483"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:34:59.966086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.966079    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs" pod "329a6854-eaf7-4e3f-b080-ad6239cde483" (UID: "329a6854-eaf7-4e3f-b080-ad6239cde483"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:34:59.966225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:34:59.966086    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6" pod "329a6854-eaf7-4e3f-b080-ad6239cde483" (UID: "329a6854-eaf7-4e3f-b080-ad6239cde483"). InnerVolumeSpecName "kube-api-access-fkzh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:00.064834 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.064742    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/329a6854-eaf7-4e3f-b080-ad6239cde483-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:00.064834 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.064774    5042 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/329a6854-eaf7-4e3f-b080-ad6239cde483-dshm\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:00.064834 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.064785    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkzh6\" (UniqueName: \"kubernetes.io/projected/329a6854-eaf7-4e3f-b080-ad6239cde483-kube-api-access-fkzh6\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:00.204702 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.204653    5042 generic.go:356] "Generic (PLEG): container finished" podID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerID="b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90" exitCode=0
Nov 02 18:35:00.204702 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.204699    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"
Nov 02 18:35:00.204991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.204741    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" event={"ID":"831a619e-ccd4-40b5-9903-e83f3daca0c8","Type":"ContainerDied","Data":"b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90"}
Nov 02 18:35:00.204991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.204789    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" event={"ID":"831a619e-ccd4-40b5-9903-e83f3daca0c8","Type":"ContainerDied","Data":"abd8824d21b4ba508016b092449e037908dd2ed6ea52f22c7f911d0dc7a1be0d"}
Nov 02 18:35:00.204991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.204813    5042 scope.go:122] "RemoveContainer" containerID="b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90"
Nov 02 18:35:00.206503 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.206481    5042 generic.go:356] "Generic (PLEG): container finished" podID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerID="c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3" exitCode=0
Nov 02 18:35:00.206612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.206526    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"
Nov 02 18:35:00.206612 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.206568    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" event={"ID":"329a6854-eaf7-4e3f-b080-ad6239cde483","Type":"ContainerDied","Data":"c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3"}
Nov 02 18:35:00.206711 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.206611    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x" event={"ID":"329a6854-eaf7-4e3f-b080-ad6239cde483","Type":"ContainerDied","Data":"9fa687fadfd6b431604ee75561ea84ed3e0d05a2bce0da5cfc1e267c7f85f59e"}
Nov 02 18:35:00.218097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.218074    5042 scope.go:122] "RemoveContainer" containerID="b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90"
Nov 02 18:35:00.218446 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:00.218425    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90\": container with ID starting with b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90 not found: ID does not exist" containerID="b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90"
Nov 02 18:35:00.218522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.218454    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90"} err="failed to get container status \"b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90\": rpc error: code = NotFound desc = could not find container \"b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90\": container with ID starting with b5f3ee3865d52037335026359424e9085f24c22e51cf744edfe6a276533e3d90 not found: ID does not exist"
Nov 02 18:35:00.218522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.218472    5042 scope.go:122] "RemoveContainer" containerID="c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3"
Nov 02 18:35:00.233034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.232327    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"]
Nov 02 18:35:00.233034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.232541    5042 scope.go:122] "RemoveContainer" containerID="c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3"
Nov 02 18:35:00.233376 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:00.233304    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3\": container with ID starting with c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3 not found: ID does not exist" containerID="c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3"
Nov 02 18:35:00.233376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.233345    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3"} err="failed to get container status \"c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3\": rpc error: code = NotFound desc = could not find container \"c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3\": container with ID starting with c11732f63af34c8709d29e8f3490093a41a1e28b6dcff091e7a4f39c8bd409e3 not found: ID does not exist"
Nov 02 18:35:00.236505 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.236484    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw"]
Nov 02 18:35:00.247607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.247580    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"]
Nov 02 18:35:00.251436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.251416    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-77f103/simulator-w4-77f103-kserve-f5f7dffc7-wbz7x"]
Nov 02 18:35:00.774255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:00.774161    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-77f103/premium-w4-77f103-kserve-5ccfbc56fb-vtxvw" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:35:01.542858 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:01.542817    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" path="/var/lib/kubelet/pods/329a6854-eaf7-4e3f-b080-ad6239cde483/volumes"
Nov 02 18:35:01.543280 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:01.543266    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" path="/var/lib/kubelet/pods/831a619e-ccd4-40b5-9903-e83f3daca0c8/volumes"
Nov 02 18:35:04.023228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:04.023191    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"]
Nov 02 18:35:04.029257 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:04.029225    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac851bcf-f7smz"]
Nov 02 18:35:05.542098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:05.542065    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0549c9-4136-492d-a11a-c98e27837baf" path="/var/lib/kubelet/pods/4f0549c9-4136-492d-a11a-c98e27837baf/volumes"
Nov 02 18:35:16.025787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:16.025729    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"]
Nov 02 18:35:16.028443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:16.028418    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3310df55-6fmjb"]
Nov 02 18:35:17.542880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:17.542826    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765282cd-b901-4a14-b250-20429c82b51d" path="/var/lib/kubelet/pods/765282cd-b901-4a14-b250-20429c82b51d/volumes"
Nov 02 18:35:18.539054 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:18.539017    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:35:24.111753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:24.111711    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"]
Nov 02 18:35:24.112372 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:24.111995    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy" containerID="cri-o://e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23" gracePeriod=30
Nov 02 18:35:25.082576 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:25.082543    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fb64bd955-5m6ww_25e41b3c-1fd8-422b-aab6-a660c5b76e4e/authorino/0.log"
Nov 02 18:35:25.281196 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:25.281158    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-lx6js_06bdf526-bccd-4303-b9f5-ff3131a4ae96/manager/0.log"
Nov 02 18:35:25.459021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:25.458985    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-k9mfj_d56437ee-40bb-439d-812f-1a55863eac69/manager/0.log"
Nov 02 18:35:25.877516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:25.877420    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f56vl_588a37ce-4b38-4224-9f67-adb7273f76b5/manager/0.log"
Nov 02 18:35:26.272788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:26.272754    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ljs7q_5488c1fe-6a3d-464a-bd15-2377b7740b9b/manager/0.log"
Nov 02 18:35:26.419258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:26.419222    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"]
Nov 02 18:35:26.419742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:26.419518    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy" containerID="cri-o://8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3" gracePeriod=30
Nov 02 18:35:27.563343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:27.563310    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/1.log"
Nov 02 18:35:29.385341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.385315    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 18:35:29.402541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.402468    5042 generic.go:356] "Generic (PLEG): container finished" podID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerID="e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23" exitCode=0
Nov 02 18:35:29.402541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.402528    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"
Nov 02 18:35:29.402777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.402555    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" event={"ID":"f0586d93-c4b4-4d5c-8df3-28f87b9c259f","Type":"ContainerDied","Data":"e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23"}
Nov 02 18:35:29.402777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.402618    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2" event={"ID":"f0586d93-c4b4-4d5c-8df3-28f87b9c259f","Type":"ContainerDied","Data":"16e86be280481f88e3c61e8cc76496101c81d4ea3539fd28a62f7e4d4e62d206"}
Nov 02 18:35:29.402777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.402650    5042 scope.go:122] "RemoveContainer" containerID="e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23"
Nov 02 18:35:29.415597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.415577    5042 scope.go:122] "RemoveContainer" containerID="e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23"
Nov 02 18:35:29.415995 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:29.415971    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23\": container with ID starting with e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23 not found: ID does not exist" containerID="e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23"
Nov 02 18:35:29.416128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.416008    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23"} err="failed to get container status \"e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23\": rpc error: code = NotFound desc = could not find container \"e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23\": container with ID starting with e30cba85aa0072c02f9c0328b0896d055dc7c26c408d9b06417b2d968680dd23 not found: ID does not exist"
Nov 02 18:35:29.481785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.481743    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.481793    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.481935    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482011 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.481989    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482026    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482059    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482091    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482134    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482162    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:29.482210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482168    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo\") pod \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\" (UID: \"f0586d93-c4b4-4d5c-8df3-28f87b9c259f\") "
Nov 02 18:35:29.482489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482220    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:35:29.482657 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482634    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.482925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482664    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.482925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.482750    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:29.483063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.483007    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:29.483120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.483081    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:29.484536 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.484511    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:29.484646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.484534    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "kube-api-access-nsdzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:29.484646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.484514    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:29.484733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.484708    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo" pod "f0586d93-c4b4-4d5c-8df3-28f87b9c259f" (UID: "f0586d93-c4b4-4d5c-8df3-28f87b9c259f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:35:29.584258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584227    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.584258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584256    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.584258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584266    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.584543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584275    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.584543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584284    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdzq\" (UniqueName: \"kubernetes.io/projected/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-kube-api-access-nsdzq\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.584543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584292    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.584543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.584300    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0586d93-c4b4-4d5c-8df3-28f87b9c259f-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:29.619236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.619208    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/0.log"
Nov 02 18:35:29.721880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.721828    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"]
Nov 02 18:35:29.730041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:29.728887    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-77f103-openshift-default-9886c5d7c-dj6t2"]
Nov 02 18:35:31.544570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.544530    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" path="/var/lib/kubelet/pods/f0586d93-c4b4-4d5c-8df3-28f87b9c259f/volumes"
Nov 02 18:35:31.682093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.682065    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:35:31.808567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808485    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.808567 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808535    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.808776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808570    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.808776 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808742    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.808944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808801    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.808944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808868    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.809071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808970    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.809071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.808972    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:35:31.809071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.809041    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.809217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.809076    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data\") pod \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\" (UID: \"7a691c30-4a84-4c9d-af0f-8bf46481d9d8\") "
Nov 02 18:35:31.809217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.809143    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:31.809380 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.809357    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:31.809766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.809589    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:31.809766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.809626    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:31.811405 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.811378    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "kube-api-access-59296". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:31.812228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.812133    5042 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istiod-ca-cert\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.812228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.812166    5042 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-credential-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.812228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.812182    5042 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-socket\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.812685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.812455    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:35:31.812685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.812500    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:35:31.813686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.813662    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token" pod "7a691c30-4a84-4c9d-af0f-8bf46481d9d8" (UID: "7a691c30-4a84-4c9d-af0f-8bf46481d9d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:31.913379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.913332    5042 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-workload-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.913379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.913370    5042 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-data\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.913379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.913385    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59296\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-kube-api-access-59296\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.913658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.913398    5042 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-podinfo\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.913658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.913410    5042 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-token\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:31.913658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:31.913421    5042 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a691c30-4a84-4c9d-af0f-8bf46481d9d8-istio-envoy\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:32.424143 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.424109    5042 generic.go:356] "Generic (PLEG): container finished" podID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerID="8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3" exitCode=0
Nov 02 18:35:32.424334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.424160    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"
Nov 02 18:35:32.424334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.424186    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" event={"ID":"7a691c30-4a84-4c9d-af0f-8bf46481d9d8","Type":"ContainerDied","Data":"8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3"}
Nov 02 18:35:32.424334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.424240    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4" event={"ID":"7a691c30-4a84-4c9d-af0f-8bf46481d9d8","Type":"ContainerDied","Data":"ea9e340de2b43ff3f73b1f9ceff7e689a4110d352857fb1ee82570bc906a3b24"}
Nov 02 18:35:32.424334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.424264    5042 scope.go:122] "RemoveContainer" containerID="8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3"
Nov 02 18:35:32.439674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.439650    5042 scope.go:122] "RemoveContainer" containerID="8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3"
Nov 02 18:35:32.440022 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:32.440002    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3\": container with ID starting with 8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3 not found: ID does not exist" containerID="8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3"
Nov 02 18:35:32.440110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.440032    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3"} err="failed to get container status \"8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3\": rpc error: code = NotFound desc = could not find container \"8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3\": container with ID starting with 8a47a9815f7c6c04aa0aa2cedbc9a5533ebb4afdd26a4311e2d9f67d8a0e09f3 not found: ID does not exist"
Nov 02 18:35:32.449676 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.449650    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"]
Nov 02 18:35:32.453777 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:32.453754    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-cmqs4"]
Nov 02 18:35:33.368707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:33.368674    5042 scope.go:122] "RemoveContainer" containerID="7244801d4e08418d5740d7891fbf2a04179c5996fd3a5322071cc8af503ccae9"
Nov 02 18:35:33.378496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:33.378476    5042 scope.go:122] "RemoveContainer" containerID="dcb5189f9da0eb7aaf1a1e6548efdf19e31ff5a5ba9c7037abf14186457f21f5"
Nov 02 18:35:33.387731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:33.387710    5042 scope.go:122] "RemoveContainer" containerID="caaedea5e614e6b6509fc130bcb3603ede5c41b5341dfdec7fd56029e3f74b4e"
Nov 02 18:35:33.543060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:33.543025    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" path="/var/lib/kubelet/pods/7a691c30-4a84-4c9d-af0f-8bf46481d9d8/volumes"
Nov 02 18:35:37.482569 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.482529    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"]
Nov 02 18:35:37.483241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483216    5042 cpu_manager.go:414] "containerMap: removing container" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main"
Nov 02 18:35:37.483241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483240    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483257    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483265    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483283    5042 cpu_manager.go:414] "containerMap: removing container" podUID="78d07225-594c-4799-b7ca-6c7290cc9202" containerName="discovery"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483291    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d07225-594c-4799-b7ca-6c7290cc9202" containerName="discovery"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483332    5042 cpu_manager.go:414] "containerMap: removing container" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483341    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483351    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483358    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483380    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" containerName="revoke-keys"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483387    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" containerName="revoke-keys"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483397    5042 cpu_manager.go:414] "containerMap: removing container" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy"
Nov 02 18:35:37.483472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483404    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483489    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="329a6854-eaf7-4e3f-b080-ad6239cde483" containerName="main"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483502    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a691c30-4a84-4c9d-af0f-8bf46481d9d8" containerName="istio-proxy"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483513    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" containerName="revoke-keys"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483524    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0586d93-c4b4-4d5c-8df3-28f87b9c259f" containerName="istio-proxy"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483532    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="831a619e-ccd4-40b5-9903-e83f3daca0c8" containerName="main"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483541    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c0ac11-411b-487c-afd9-abe76d467bba" containerName="payload-processing"
Nov 02 18:35:37.484231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.483550    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d07225-594c-4799-b7ca-6c7290cc9202" containerName="discovery"
Nov 02 18:35:37.489265 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.489240    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.495885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.495819    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"]
Nov 02 18:35:37.519948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.519924    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f8e1abfd-dockercfg-q5s4t\""
Nov 02 18:35:37.559024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.558978    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f8e1abfd\""
Nov 02 18:35:37.570593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.570548    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.570877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.570812    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwvw\" (UniqueName: \"kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw\") pod \"payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.672379 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.672342    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.672602 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.672500    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwvw\" (UniqueName: \"kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw\") pod \"payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.673164 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.673142    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume\") pod \"payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.683151 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.683127    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwvw\" (UniqueName: \"kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw\") pod \"payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") " pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:37.808252 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:37.808163    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:38.025650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.025606    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"]
Nov 02 18:35:38.028298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.028272    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c3245f-zmhch"]
Nov 02 18:35:38.097653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.097627    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"]
Nov 02 18:35:38.100772 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:35:38.100732    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3962089_f3b9_4a22_949d_434994f91fd9.slice/crio-beb894053c3bb541ca1aed4eb829be8db8ca54376b7716ab0725c2a9438d8d48 WatchSource:0}: Error finding container beb894053c3bb541ca1aed4eb829be8db8ca54376b7716ab0725c2a9438d8d48: Status 404 returned error can't find the container with id beb894053c3bb541ca1aed4eb829be8db8ca54376b7716ab0725c2a9438d8d48
Nov 02 18:35:38.469937 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.469899    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk" event={"ID":"e3962089-f3b9-4a22-949d-434994f91fd9","Type":"ContainerStarted","Data":"b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54"}
Nov 02 18:35:38.470171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.469958    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk" event={"ID":"e3962089-f3b9-4a22-949d-434994f91fd9","Type":"ContainerStarted","Data":"beb894053c3bb541ca1aed4eb829be8db8ca54376b7716ab0725c2a9438d8d48"}
Nov 02 18:35:38.470171 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.470115    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:38.485860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:38.485453    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk" podStartSLOduration=1.485435697 podStartE2EDuration="1.485435697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:35:38.484977858 +0000 UTC m=+3751.064387255" watchObservedRunningTime="2026-11-02 18:35:38.485435697 +0000 UTC m=+3751.064845084"
Nov 02 18:35:39.552646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:39.552605    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e75f8abb-f63e-4d4f-947f-399cf2748150" path="/var/lib/kubelet/pods/e75f8abb-f63e-4d4f-947f-399cf2748150/volumes"
Nov 02 18:35:48.538447 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:48.538408    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:35:49.482146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:49.482117    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:35:50.022039 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:50.022000    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"]
Nov 02 18:35:50.024526 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:50.024502    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70121e82-2kmv6"]
Nov 02 18:35:51.542640 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:51.542605    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83" path="/var/lib/kubelet/pods/c12bbf0b-c06f-4d94-831b-1b4d4a3a1c83/volumes"
Nov 02 18:35:52.026158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.026109    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"]
Nov 02 18:35:52.030283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.030254    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"]
Nov 02 18:35:52.030502 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.030399    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.033592 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.033567    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ebc8f8-g6dtd"]
Nov 02 18:35:52.034972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.034947    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:35:52.035160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.034992    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:35:52.035160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.035083    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b26tc\""
Nov 02 18:35:52.035160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.035150    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:35:52.036717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.036694    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"]
Nov 02 18:35:52.119532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.119486    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.119741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.119664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.119741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.119718    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.220289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.220246    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.220480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.220350    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.220480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.220387    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.221021 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.220996    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.222907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.222890    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.226280 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:52.226249    5042 projected.go:196] Error preparing data for projected volume kube-api-access-7fnvd for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 18:35:52.226393 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:52.226334    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd podName:6f7296f1-a9e7-498c-b4df-8a851d6f09bf nodeName:}" failed. No retries permitted until 2026-11-02 18:35:52.726319556 +0000 UTC m=+3765.305728923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7fnvd" (UniqueName: "kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd") pod "maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8" (UID: "6f7296f1-a9e7-498c-b4df-8a851d6f09bf") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 18:35:52.826499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:52.826463    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:52.829978 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:52.829951    5042 projected.go:196] Error preparing data for projected volume kube-api-access-7fnvd for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 18:35:52.830154 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:35:52.830041    5042 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd podName:6f7296f1-a9e7-498c-b4df-8a851d6f09bf nodeName:}" failed. No retries permitted until 2026-11-02 18:35:53.830020849 +0000 UTC m=+3766.409430228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7fnvd" (UniqueName: "kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd") pod "maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8" (UID: "6f7296f1-a9e7-498c-b4df-8a851d6f09bf") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 18:35:53.542707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:53.542673    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604905ae-308b-4d5f-8def-7c9ecbf28d41" path="/var/lib/kubelet/pods/604905ae-308b-4d5f-8def-7c9ecbf28d41/volumes"
Nov 02 18:35:53.836794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:53.836694    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:53.847988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:53.847958    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") pod \"maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:54.147304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:54.147211    5042 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8" secret="" err="secret \"maas-api-cleanup-dockercfg-b26tc\" not found"
Nov 02 18:35:54.147304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:54.147297    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:54.426643 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:54.426617    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"]
Nov 02 18:35:54.428756 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:35:54.428726    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7296f1_a9e7_498c_b4df_8a851d6f09bf.slice/crio-2c0f1a00abf2b6a9fa9fc84afdc9c97e164adc0fc86e8955cb367bf3fe95c456 WatchSource:0}: Error finding container 2c0f1a00abf2b6a9fa9fc84afdc9c97e164adc0fc86e8955cb367bf3fe95c456: Status 404 returned error can't find the container with id 2c0f1a00abf2b6a9fa9fc84afdc9c97e164adc0fc86e8955cb367bf3fe95c456
Nov 02 18:35:54.573636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:54.573603    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8" event={"ID":"6f7296f1-a9e7-498c-b4df-8a851d6f09bf","Type":"ContainerStarted","Data":"2c0f1a00abf2b6a9fa9fc84afdc9c97e164adc0fc86e8955cb367bf3fe95c456"}
Nov 02 18:35:55.582116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:55.582024    5042 generic.go:356] "Generic (PLEG): container finished" podID="6f7296f1-a9e7-498c-b4df-8a851d6f09bf" containerID="f0b4039bb0d8c0927314c0f00262886d39f41e5286e6a757c9acea6c717017fb" exitCode=0
Nov 02 18:35:55.582116 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:55.582062    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8" event={"ID":"6f7296f1-a9e7-498c-b4df-8a851d6f09bf","Type":"ContainerDied","Data":"f0b4039bb0d8c0927314c0f00262886d39f41e5286e6a757c9acea6c717017fb"}
Nov 02 18:35:56.731330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.731293    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:56.767094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.767052    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") pod \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") "
Nov 02 18:35:56.767094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.767099    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca\") pod \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") "
Nov 02 18:35:56.767380 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.767180    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca\") pod \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\" (UID: \"6f7296f1-a9e7-498c-b4df-8a851d6f09bf\") "
Nov 02 18:35:56.767546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.767479    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca" pod "6f7296f1-a9e7-498c-b4df-8a851d6f09bf" (UID: "6f7296f1-a9e7-498c-b4df-8a851d6f09bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:35:56.767671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.767653    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:56.769448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.769419    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca" pod "6f7296f1-a9e7-498c-b4df-8a851d6f09bf" (UID: "6f7296f1-a9e7-498c-b4df-8a851d6f09bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:35:56.769605 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.769581    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd" pod "6f7296f1-a9e7-498c-b4df-8a851d6f09bf" (UID: "6f7296f1-a9e7-498c-b4df-8a851d6f09bf"). InnerVolumeSpecName "kube-api-access-7fnvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:35:56.869080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.868993    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fnvd\" (UniqueName: \"kubernetes.io/projected/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-kube-api-access-7fnvd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:56.869080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:56.869024    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f7296f1-a9e7-498c-b4df-8a851d6f09bf-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:35:57.596555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:57.596524    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"
Nov 02 18:35:57.596555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:57.596541    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8" event={"ID":"6f7296f1-a9e7-498c-b4df-8a851d6f09bf","Type":"ContainerDied","Data":"2c0f1a00abf2b6a9fa9fc84afdc9c97e164adc0fc86e8955cb367bf3fe95c456"}
Nov 02 18:35:57.596787 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:35:57.596595    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0f1a00abf2b6a9fa9fc84afdc9c97e164adc0fc86e8955cb367bf3fe95c456"
Nov 02 18:36:02.155086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:02.155046    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"]
Nov 02 18:36:03.029301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:03.029254    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"]
Nov 02 18:36:03.031853 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:03.031807    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8bc5e9e-wb2wz"]
Nov 02 18:36:03.167629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:03.167590    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"]
Nov 02 18:36:03.169986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:03.169963    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f8e1abfd-22mt8"]
Nov 02 18:36:03.543053 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:03.543021    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7296f1-a9e7-498c-b4df-8a851d6f09bf" path="/var/lib/kubelet/pods/6f7296f1-a9e7-498c-b4df-8a851d6f09bf/volumes"
Nov 02 18:36:03.543428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:03.543415    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf7aa2b-f16a-48c6-8df1-44b8a115d530" path="/var/lib/kubelet/pods/abf7aa2b-f16a-48c6-8df1-44b8a115d530/volumes"
Nov 02 18:36:07.155747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.155698    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk" podUID="e3962089-f3b9-4a22-949d-434994f91fd9" containerName="payload-processing" containerID="cri-o://b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54" gracePeriod=25
Nov 02 18:36:07.436806 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.436771    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:36:07.582995 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.582960    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw\" (UniqueName: \"kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw\") pod \"e3962089-f3b9-4a22-949d-434994f91fd9\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") "
Nov 02 18:36:07.583155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.583011    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume\") pod \"e3962089-f3b9-4a22-949d-434994f91fd9\" (UID: \"e3962089-f3b9-4a22-949d-434994f91fd9\") "
Nov 02 18:36:07.583457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.583433    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume" pod "e3962089-f3b9-4a22-949d-434994f91fd9" (UID: "e3962089-f3b9-4a22-949d-434994f91fd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:07.585443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.585412    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw" pod "e3962089-f3b9-4a22-949d-434994f91fd9" (UID: "e3962089-f3b9-4a22-949d-434994f91fd9"). InnerVolumeSpecName "kube-api-access-mcwvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:07.674176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.674086    5042 generic.go:356] "Generic (PLEG): container finished" podID="e3962089-f3b9-4a22-949d-434994f91fd9" containerID="b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54" exitCode=0
Nov 02 18:36:07.674176 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.674151    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"
Nov 02 18:36:07.674383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.674178    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk" event={"ID":"e3962089-f3b9-4a22-949d-434994f91fd9","Type":"ContainerDied","Data":"b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54"}
Nov 02 18:36:07.674383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.674239    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk" event={"ID":"e3962089-f3b9-4a22-949d-434994f91fd9","Type":"ContainerDied","Data":"beb894053c3bb541ca1aed4eb829be8db8ca54376b7716ab0725c2a9438d8d48"}
Nov 02 18:36:07.674383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.674265    5042 scope.go:122] "RemoveContainer" containerID="b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54"
Nov 02 18:36:07.684058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.684034    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwvw\" (UniqueName: \"kubernetes.io/projected/e3962089-f3b9-4a22-949d-434994f91fd9-kube-api-access-mcwvw\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:36:07.684058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.684057    5042 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3962089-f3b9-4a22-949d-434994f91fd9-plugins-config-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:36:07.684249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.684221    5042 scope.go:122] "RemoveContainer" containerID="b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54"
Nov 02 18:36:07.684544 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:36:07.684520    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54\": container with ID starting with b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54 not found: ID does not exist" containerID="b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54"
Nov 02 18:36:07.684623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.684554    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54"} err="failed to get container status \"b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54\": rpc error: code = NotFound desc = could not find container \"b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54\": container with ID starting with b183343b6aecbd787fe9f5a0b6ce662b4fc0de1c1f2b52d281555bea91dd9e54 not found: ID does not exist"
Nov 02 18:36:07.701482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.701453    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"]
Nov 02 18:36:07.707199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:07.707171    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f8e1abfd-677b84899c-xcjjk"]
Nov 02 18:36:09.542099 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:09.542070    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3962089-f3b9-4a22-949d-434994f91fd9" path="/var/lib/kubelet/pods/e3962089-f3b9-4a22-949d-434994f91fd9/volumes"
Nov 02 18:36:18.539483 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:36:18.539439    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:36:26.539796 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.539759    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"]
Nov 02 18:36:26.540448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.540364    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6f7296f1-a9e7-498c-b4df-8a851d6f09bf" containerName="revoke-keys"
Nov 02 18:36:26.540448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.540380    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7296f1-a9e7-498c-b4df-8a851d6f09bf" containerName="revoke-keys"
Nov 02 18:36:26.540448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.540389    5042 cpu_manager.go:414] "containerMap: removing container" podUID="e3962089-f3b9-4a22-949d-434994f91fd9" containerName="payload-processing"
Nov 02 18:36:26.540448 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.540394    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3962089-f3b9-4a22-949d-434994f91fd9" containerName="payload-processing"
Nov 02 18:36:26.540686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.540508    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7296f1-a9e7-498c-b4df-8a851d6f09bf" containerName="revoke-keys"
Nov 02 18:36:26.540686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.540518    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3962089-f3b9-4a22-949d-434994f91fd9" containerName="payload-processing"
Nov 02 18:36:26.544118 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.544094    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.558905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.558878    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"]
Nov 02 18:36:26.574260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574208    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574268    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574457 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574295    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5557b2eb-4759-40b7-b6f2-c41494af6815-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574460    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574581    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574648    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574736    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574781    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.574960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.574904    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxlm\" (UniqueName: \"kubernetes.io/projected/5557b2eb-4759-40b7-b6f2-c41494af6815-kube-api-access-6pxlm\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676432    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxlm\" (UniqueName: \"kubernetes.io/projected/5557b2eb-4759-40b7-b6f2-c41494af6815-kube-api-access-6pxlm\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676507    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676534    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676556    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5557b2eb-4759-40b7-b6f2-c41494af6815-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676588    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676619    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.676678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676658    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676728    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.676765    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.677044    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-data\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.677144    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-workload-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.677209    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-credential-socket\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677329 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.677308    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-workload-certs\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.677515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.677497    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5557b2eb-4759-40b7-b6f2-c41494af6815-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.679150 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.679131    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-envoy\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.679414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.679394    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-podinfo\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.684664 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.684636    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5557b2eb-4759-40b7-b6f2-c41494af6815-istio-token\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.684810 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.684765    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxlm\" (UniqueName: \"kubernetes.io/projected/5557b2eb-4759-40b7-b6f2-c41494af6815-kube-api-access-6pxlm\") pod \"maas-default-gateway-openshift-default-65c448b547-zws9j\" (UID: \"5557b2eb-4759-40b7-b6f2-c41494af6815\") " pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:26.860721 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:26.860629    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:27.141687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.141656    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"]
Nov 02 18:36:27.143164 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:36:27.143140    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5557b2eb_4759_40b7_b6f2_c41494af6815.slice/crio-245ecf123e82980e1b3ad33afc72d4cb54247ba24c59e74dc70e338093c1254c WatchSource:0}: Error finding container 245ecf123e82980e1b3ad33afc72d4cb54247ba24c59e74dc70e338093c1254c: Status 404 returned error can't find the container with id 245ecf123e82980e1b3ad33afc72d4cb54247ba24c59e74dc70e338093c1254c
Nov 02 18:36:27.145614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.145575    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:36:27.145719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.145700    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:36:27.145765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.145751    5042 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:36:27.807440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.807400    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" event={"ID":"5557b2eb-4759-40b7-b6f2-c41494af6815","Type":"ContainerStarted","Data":"c9c268e42ecf2c7795dcb1f4e01e579174d34dc61822f665d81e5b2dd1a47362"}
Nov 02 18:36:27.808010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.807459    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" event={"ID":"5557b2eb-4759-40b7-b6f2-c41494af6815","Type":"ContainerStarted","Data":"245ecf123e82980e1b3ad33afc72d4cb54247ba24c59e74dc70e338093c1254c"}
Nov 02 18:36:27.829083 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.829034    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podStartSLOduration=1.8290189350000001 podStartE2EDuration="1.829018935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:27.826367195 +0000 UTC m=+3800.405776582" watchObservedRunningTime="2026-11-02 18:36:27.829018935 +0000 UTC m=+3800.408428320"
Nov 02 18:36:27.860980 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:27.860948    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:28.862542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:28.862505    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 02 18:36:28.863000 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:28.862561    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 02 18:36:29.862028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:29.861988    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 02 18:36:29.862234 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:29.862049    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 02 18:36:30.861018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:30.860981    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 02 18:36:30.861454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:30.861039    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 02 18:36:31.475868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.475327    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"]
Nov 02 18:36:31.481861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.481209    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.494934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.494903    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:36:31.495238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.495221    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:36:31.495464 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.495447    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:36:31.495813 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.495795    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:36:31.498008 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.497975    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"]
Nov 02 18:36:31.527683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.527644    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.527968 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.527947    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.528111 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.528096    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd26f\" (UniqueName: \"kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.629418 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.629373    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.629627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.629425    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.629627 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.629473    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd26f\" (UniqueName: \"kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.630665 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.630640    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.632477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.632453    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.637881 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.637856    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd26f\" (UniqueName: \"kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f\") pod \"maas-api-revoke-sub-85237e45-ctbbt\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.818811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.818722    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:31.861949 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.861909    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 02 18:36:31.862429 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:31.861965    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 02 18:36:32.125551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:32.125518    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"]
Nov 02 18:36:32.127612 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:36:32.127585    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127aaba2_9099_4be8_b192_4ded210aea33.slice/crio-7f3fc5b2ac8733948bed53d6580d3b9fba2d916da35ab681c9cb8e3d7b9bed22 WatchSource:0}: Error finding container 7f3fc5b2ac8733948bed53d6580d3b9fba2d916da35ab681c9cb8e3d7b9bed22: Status 404 returned error can't find the container with id 7f3fc5b2ac8733948bed53d6580d3b9fba2d916da35ab681c9cb8e3d7b9bed22
Nov 02 18:36:32.843373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:32.843333    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt" event={"ID":"127aaba2-9099-4be8-b192-4ded210aea33","Type":"ContainerStarted","Data":"5af4c57bddfed7f2f58b66547f27f75ee217f5f250722c19d2fce31bcf07cce2"}
Nov 02 18:36:32.843537 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:32.843394    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt" event={"ID":"127aaba2-9099-4be8-b192-4ded210aea33","Type":"ContainerStarted","Data":"7f3fc5b2ac8733948bed53d6580d3b9fba2d916da35ab681c9cb8e3d7b9bed22"}
Nov 02 18:36:32.861714 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:32.861680    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 02 18:36:32.861950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:32.861742    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 02 18:36:32.862485 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:32.862442    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt" podStartSLOduration=1.862428372 podStartE2EDuration="1.862428372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:36:32.859989219 +0000 UTC m=+3805.439398605" watchObservedRunningTime="2026-11-02 18:36:32.862428372 +0000 UTC m=+3805.441837795"
Nov 02 18:36:33.456753 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.456721    5042 scope.go:122] "RemoveContainer" containerID="3a956d592af8193cc3328952016769c9d8f49f92873a7bb5cf068964fbc58763"
Nov 02 18:36:33.467256 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.467231    5042 scope.go:122] "RemoveContainer" containerID="f1a2a8a4beedf1aeadbe9db0fba1699305fbd5027f0a409e277b5061ddeb1db1"
Nov 02 18:36:33.476940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.476916    5042 scope.go:122] "RemoveContainer" containerID="b49f4aeec5a4665f244f5eebb486913e462eaa99f098087e480ea96f632370bb"
Nov 02 18:36:33.855239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.855140    5042 generic.go:356] "Generic (PLEG): container finished" podID="127aaba2-9099-4be8-b192-4ded210aea33" containerID="5af4c57bddfed7f2f58b66547f27f75ee217f5f250722c19d2fce31bcf07cce2" exitCode=0
Nov 02 18:36:33.855239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.855193    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt" event={"ID":"127aaba2-9099-4be8-b192-4ded210aea33","Type":"ContainerDied","Data":"5af4c57bddfed7f2f58b66547f27f75ee217f5f250722c19d2fce31bcf07cce2"}
Nov 02 18:36:33.861423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.861385    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused" start-of-body=
Nov 02 18:36:33.861611 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:33.861449    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": dial tcp 10.132.1.119:15021: connect: connection refused"
Nov 02 18:36:35.023200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.023156    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:35.066440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.066398    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca\") pod \"127aaba2-9099-4be8-b192-4ded210aea33\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") "
Nov 02 18:36:35.066660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.066507    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f\" (UniqueName: \"kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f\") pod \"127aaba2-9099-4be8-b192-4ded210aea33\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") "
Nov 02 18:36:35.066660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.066574    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca\") pod \"127aaba2-9099-4be8-b192-4ded210aea33\" (UID: \"127aaba2-9099-4be8-b192-4ded210aea33\") "
Nov 02 18:36:35.067195 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.067154    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca" pod "127aaba2-9099-4be8-b192-4ded210aea33" (UID: "127aaba2-9099-4be8-b192-4ded210aea33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:36:35.068762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.068739    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca" pod "127aaba2-9099-4be8-b192-4ded210aea33" (UID: "127aaba2-9099-4be8-b192-4ded210aea33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:36:35.069322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.069303    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f" pod "127aaba2-9099-4be8-b192-4ded210aea33" (UID: "127aaba2-9099-4be8-b192-4ded210aea33"). InnerVolumeSpecName "kube-api-access-xd26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:36:35.168604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.168508    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd26f\" (UniqueName: \"kubernetes.io/projected/127aaba2-9099-4be8-b192-4ded210aea33-kube-api-access-xd26f\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:36:35.168604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.168554    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:36:35.168604 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.168571    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/127aaba2-9099-4be8-b192-4ded210aea33-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:36:35.862255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.862148    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:36:35.862255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.862232    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:36:35.869329 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.869292    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt" event={"ID":"127aaba2-9099-4be8-b192-4ded210aea33","Type":"ContainerDied","Data":"7f3fc5b2ac8733948bed53d6580d3b9fba2d916da35ab681c9cb8e3d7b9bed22"}
Nov 02 18:36:35.869329 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.869309    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"
Nov 02 18:36:35.869594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:35.869344    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3fc5b2ac8733948bed53d6580d3b9fba2d916da35ab681c9cb8e3d7b9bed22"
Nov 02 18:36:36.903081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:36.903040    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:36:36.903614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:36.903116    5042 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:36:37.597287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:37.597257    5042 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:37.597822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:37.597795    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:37.598622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:37.598603    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j"
Nov 02 18:36:48.540201 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:36:48.540120    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:36:57.024248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:57.024209    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"]
Nov 02 18:36:57.027137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:57.027104    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f33f80c-94nxm"]
Nov 02 18:36:57.543298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:36:57.543255    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24fa48a8-767e-434d-abb1-f99968544a75" path="/var/lib/kubelet/pods/24fa48a8-767e-434d-abb1-f99968544a75/volumes"
Nov 02 18:37:07.032310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:07.032264    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"]
Nov 02 18:37:07.069924 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:07.069881    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-326225ac-mbdrz"]
Nov 02 18:37:07.556504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:07.556457    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329c5b33-9bf4-4505-a3d1-5ff44ebf248c" path="/var/lib/kubelet/pods/329c5b33-9bf4-4505-a3d1-5ff44ebf248c/volumes"
Nov 02 18:37:09.693613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.693572    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"]
Nov 02 18:37:09.694070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.694034    5042 cpu_manager.go:414] "containerMap: removing container" podUID="127aaba2-9099-4be8-b192-4ded210aea33" containerName="revoke-keys"
Nov 02 18:37:09.694070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.694046    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127aaba2-9099-4be8-b192-4ded210aea33" containerName="revoke-keys"
Nov 02 18:37:09.694158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.694144    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="127aaba2-9099-4be8-b192-4ded210aea33" containerName="revoke-keys"
Nov 02 18:37:09.698888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.698866    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.704080 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.704050    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:37:09.704273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.704205    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:37:09.704273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.704243    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:37:09.704409 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.704313    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:37:09.707451 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.707410    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"]
Nov 02 18:37:09.752717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.752680    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mxb\" (UniqueName: \"kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.752946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.752815    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.752946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.752923    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.853733 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.853683    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mxb\" (UniqueName: \"kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.853977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.853774    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.853977 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.853884    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.854479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.854442    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.856554 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.856528    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:09.862590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:09.862565    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mxb\" (UniqueName: \"kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb\") pod \"maas-api-revoke-sub-f904bff9-h69n4\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:10.015617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:10.015524    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:10.294416 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:10.294390    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"]
Nov 02 18:37:10.296082 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:37:10.296048    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4fc0a9_8e92_4db4_9150_79f3047eb1dd.slice/crio-540aa541ba120b89ee48e57e0ec036e6c2b4e4b6eccd6bff6843df07d1e7986c WatchSource:0}: Error finding container 540aa541ba120b89ee48e57e0ec036e6c2b4e4b6eccd6bff6843df07d1e7986c: Status 404 returned error can't find the container with id 540aa541ba120b89ee48e57e0ec036e6c2b4e4b6eccd6bff6843df07d1e7986c
Nov 02 18:37:11.121269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:11.121231    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4" event={"ID":"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd","Type":"ContainerStarted","Data":"b0285fb680f8b882ad7a414b3a3b6b606e6396200a6b387f99872af428c159c7"}
Nov 02 18:37:11.121704 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:11.121291    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4" event={"ID":"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd","Type":"ContainerStarted","Data":"540aa541ba120b89ee48e57e0ec036e6c2b4e4b6eccd6bff6843df07d1e7986c"}
Nov 02 18:37:11.140581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:11.140522    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4" podStartSLOduration=2.140508371 podStartE2EDuration="2.140508371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:37:11.137985847 +0000 UTC m=+3843.717395233" watchObservedRunningTime="2026-11-02 18:37:11.140508371 +0000 UTC m=+3843.719917756"
Nov 02 18:37:12.131006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:12.130971    5042 generic.go:356] "Generic (PLEG): container finished" podID="dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" containerID="b0285fb680f8b882ad7a414b3a3b6b606e6396200a6b387f99872af428c159c7" exitCode=0
Nov 02 18:37:12.131390 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:12.131059    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4" event={"ID":"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd","Type":"ContainerDied","Data":"b0285fb680f8b882ad7a414b3a3b6b606e6396200a6b387f99872af428c159c7"}
Nov 02 18:37:13.291048 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.291018    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:13.389803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.389710    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca\") pod \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") "
Nov 02 18:37:13.389803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.389767    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb\" (UniqueName: \"kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb\") pod \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") "
Nov 02 18:37:13.390054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.389886    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca\") pod \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\" (UID: \"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd\") "
Nov 02 18:37:13.390144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.390122    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca" pod "dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" (UID: "dc4fc0a9-8e92-4db4-9150-79f3047eb1dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:13.392251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.392221    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca" pod "dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" (UID: "dc4fc0a9-8e92-4db4-9150-79f3047eb1dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:37:13.392365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.392274    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb" pod "dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" (UID: "dc4fc0a9-8e92-4db4-9150-79f3047eb1dd"). InnerVolumeSpecName "kube-api-access-w7mxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:13.490905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.490833    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:13.490905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.490898    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:13.490905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:13.490912    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mxb\" (UniqueName: \"kubernetes.io/projected/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd-kube-api-access-w7mxb\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:14.144811 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:14.144778    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"
Nov 02 18:37:14.145014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:14.144810    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4" event={"ID":"dc4fc0a9-8e92-4db4-9150-79f3047eb1dd","Type":"ContainerDied","Data":"540aa541ba120b89ee48e57e0ec036e6c2b4e4b6eccd6bff6843df07d1e7986c"}
Nov 02 18:37:14.145014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:14.144868    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="540aa541ba120b89ee48e57e0ec036e6c2b4e4b6eccd6bff6843df07d1e7986c"
Nov 02 18:37:18.023343 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:18.023301    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"]
Nov 02 18:37:18.026188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:18.026161    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32cc49b1-4f67x"]
Nov 02 18:37:18.538494 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:37:18.538450    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:37:19.542543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:19.542508    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c92a41f-62c2-47dd-9d80-4a34e0edbc81" path="/var/lib/kubelet/pods/1c92a41f-62c2-47dd-9d80-4a34e0edbc81/volumes"
Nov 02 18:37:29.043960 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:29.043914    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"]
Nov 02 18:37:29.046793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:29.046753    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da187c9c-m4k9g"]
Nov 02 18:37:29.553868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:29.553169    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c9377c-f1e2-457d-95f8-2e26a8463ef6" path="/var/lib/kubelet/pods/f0c9377c-f1e2-457d-95f8-2e26a8463ef6/volumes"
Nov 02 18:37:31.974756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.974717    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"]
Nov 02 18:37:31.975210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.975196    5042 cpu_manager.go:414] "containerMap: removing container" podUID="dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" containerName="revoke-keys"
Nov 02 18:37:31.975263 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.975211    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" containerName="revoke-keys"
Nov 02 18:37:31.975313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.975304    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" containerName="revoke-keys"
Nov 02 18:37:31.979783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.979765    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:31.983989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.983959    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:37:31.984145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.984113    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:37:31.984230 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.984209    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:37:31.984376 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.984325    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:37:31.986141 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:31.986119    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"]
Nov 02 18:37:32.076442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.076389    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747w9\" (UniqueName: \"kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.076630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.076520    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.076630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.076551    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.178064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.178012    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747w9\" (UniqueName: \"kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.178228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.178109    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.178228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.178132    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.178762 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.178738    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.180770 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.180749    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.187276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.187249    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-747w9\" (UniqueName: \"kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9\") pod \"maas-api-revoke-sub-f03629db-wbd8m\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.295298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.295189    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:32.577431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:32.577406    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"]
Nov 02 18:37:32.579460 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:37:32.579432    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23285ac0_1b5d_47b8_98d5_5d1eaa3e2751.slice/crio-b37ba3a00c008d4c9c6d3caa3aaa111c601d9133047d82a8dd0964513ef18a14 WatchSource:0}: Error finding container b37ba3a00c008d4c9c6d3caa3aaa111c601d9133047d82a8dd0964513ef18a14: Status 404 returned error can't find the container with id b37ba3a00c008d4c9c6d3caa3aaa111c601d9133047d82a8dd0964513ef18a14
Nov 02 18:37:33.319272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.319233    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m" event={"ID":"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751","Type":"ContainerStarted","Data":"86499bfb6332fd637ba26815ad567d623fd27c7fc9df33e47b184047a99134be"}
Nov 02 18:37:33.319694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.319346    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m" event={"ID":"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751","Type":"ContainerStarted","Data":"b37ba3a00c008d4c9c6d3caa3aaa111c601d9133047d82a8dd0964513ef18a14"}
Nov 02 18:37:33.338545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.338485    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m" podStartSLOduration=2.338472207 podStartE2EDuration="2.338472207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:37:33.336145022 +0000 UTC m=+3865.915554399" watchObservedRunningTime="2026-11-02 18:37:33.338472207 +0000 UTC m=+3865.917881592"
Nov 02 18:37:33.531824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.531788    5042 scope.go:122] "RemoveContainer" containerID="234996f67d4dc48f400325fa227ff43b188fb0e7bcc98a270b74b77526e8cd28"
Nov 02 18:37:33.542199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.542172    5042 scope.go:122] "RemoveContainer" containerID="37c1d538f752a1e7b4d4e08cdf204108f1682de656537c46e0e7deaf85620d76"
Nov 02 18:37:33.551870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.551823    5042 scope.go:122] "RemoveContainer" containerID="906386577c6f3ee86ff6a3770c401ee30c1b5e90b9080bd7d3a03e56ecf0ea77"
Nov 02 18:37:33.561165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:33.561136    5042 scope.go:122] "RemoveContainer" containerID="58eedb232cdc7c33729db8e79c8fa569fc7a3b0ebada495a2878613dd5665dee"
Nov 02 18:37:34.326565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:34.326534    5042 generic.go:356] "Generic (PLEG): container finished" podID="23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" containerID="86499bfb6332fd637ba26815ad567d623fd27c7fc9df33e47b184047a99134be" exitCode=0
Nov 02 18:37:34.327018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:34.326593    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m" event={"ID":"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751","Type":"ContainerDied","Data":"86499bfb6332fd637ba26815ad567d623fd27c7fc9df33e47b184047a99134be"}
Nov 02 18:37:35.516476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.516445    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:35.614420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.614331    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca\") pod \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") "
Nov 02 18:37:35.614420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.614389    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9\" (UniqueName: \"kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9\") pod \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") "
Nov 02 18:37:35.614678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.614457    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca\") pod \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\" (UID: \"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751\") "
Nov 02 18:37:35.614920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.614896    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca" pod "23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" (UID: "23285ac0-1b5d-47b8-98d5-5d1eaa3e2751"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:35.616710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.616686    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca" pod "23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" (UID: "23285ac0-1b5d-47b8-98d5-5d1eaa3e2751"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:37:35.616816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.616704    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9" pod "23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" (UID: "23285ac0-1b5d-47b8-98d5-5d1eaa3e2751"). InnerVolumeSpecName "kube-api-access-747w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:35.716293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.716242    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:35.716293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.716288    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:35.716293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:35.716298    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-747w9\" (UniqueName: \"kubernetes.io/projected/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751-kube-api-access-747w9\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:36.340242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:36.340207    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"
Nov 02 18:37:36.340242 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:36.340234    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m" event={"ID":"23285ac0-1b5d-47b8-98d5-5d1eaa3e2751","Type":"ContainerDied","Data":"b37ba3a00c008d4c9c6d3caa3aaa111c601d9133047d82a8dd0964513ef18a14"}
Nov 02 18:37:36.340480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:36.340285    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37ba3a00c008d4c9c6d3caa3aaa111c601d9133047d82a8dd0964513ef18a14"
Nov 02 18:37:47.841197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.841157    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"]
Nov 02 18:37:47.841955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.841872    5042 cpu_manager.go:414] "containerMap: removing container" podUID="23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" containerName="revoke-keys"
Nov 02 18:37:47.841955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.841893    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" containerName="revoke-keys"
Nov 02 18:37:47.842145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.841985    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" containerName="revoke-keys"
Nov 02 18:37:47.845292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.845271    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:47.851160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.850887    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:37:47.851160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.850973    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:37:47.851160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.851030    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:37:47.851160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.851126    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:37:47.853430 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.853400    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"]
Nov 02 18:37:47.973141 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.973104    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:47.973396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.973254    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:47.973396 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:47.973361    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66m4s\" (UniqueName: \"kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.074330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.074287    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.074551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.074413    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.074551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.074457    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66m4s\" (UniqueName: \"kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.074978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.074954    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.076943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.076921    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.082624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.082597    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66m4s\" (UniqueName: \"kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s\") pod \"maas-api-revoke-sub-0165d7d2-s2kqf\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.163212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.163129    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:48.441666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:48.441636    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"]
Nov 02 18:37:48.442590 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:37:48.442560    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a59e43_43df_4b98_8824_2df9767fa85e.slice/crio-d46f419c493e08e467f1dc2fbb12e889047ea7ef223bf31091870fdab6e0459b WatchSource:0}: Error finding container d46f419c493e08e467f1dc2fbb12e889047ea7ef223bf31091870fdab6e0459b: Status 404 returned error can't find the container with id d46f419c493e08e467f1dc2fbb12e889047ea7ef223bf31091870fdab6e0459b
Nov 02 18:37:48.538432 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:37:48.538398    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:37:49.442262 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:49.442160    5042 generic.go:356] "Generic (PLEG): container finished" podID="08a59e43-43df-4b98-8824-2df9767fa85e" containerID="99d9c091638918e405824e2950fa1b9e0e19c055fbbff6113ef743ba191ecab6" exitCode=0
Nov 02 18:37:49.442262 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:49.442252    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf" event={"ID":"08a59e43-43df-4b98-8824-2df9767fa85e","Type":"ContainerDied","Data":"99d9c091638918e405824e2950fa1b9e0e19c055fbbff6113ef743ba191ecab6"}
Nov 02 18:37:49.442741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:49.442298    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf" event={"ID":"08a59e43-43df-4b98-8824-2df9767fa85e","Type":"ContainerStarted","Data":"d46f419c493e08e467f1dc2fbb12e889047ea7ef223bf31091870fdab6e0459b"}
Nov 02 18:37:50.601907 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.601877    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:50.699548 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.699506    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s\" (UniqueName: \"kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s\") pod \"08a59e43-43df-4b98-8824-2df9767fa85e\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") "
Nov 02 18:37:50.699750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.699570    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca\") pod \"08a59e43-43df-4b98-8824-2df9767fa85e\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") "
Nov 02 18:37:50.699750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.699613    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca\") pod \"08a59e43-43df-4b98-8824-2df9767fa85e\" (UID: \"08a59e43-43df-4b98-8824-2df9767fa85e\") "
Nov 02 18:37:50.700051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.700023    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca" pod "08a59e43-43df-4b98-8824-2df9767fa85e" (UID: "08a59e43-43df-4b98-8824-2df9767fa85e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:37:50.701966 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.701931    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca" pod "08a59e43-43df-4b98-8824-2df9767fa85e" (UID: "08a59e43-43df-4b98-8824-2df9767fa85e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:37:50.702105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.702065    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s" pod "08a59e43-43df-4b98-8824-2df9767fa85e" (UID: "08a59e43-43df-4b98-8824-2df9767fa85e"). InnerVolumeSpecName "kube-api-access-66m4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:37:50.801515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.801473    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:50.801515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.801510    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08a59e43-43df-4b98-8824-2df9767fa85e-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:50.801515 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:50.801520    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66m4s\" (UniqueName: \"kubernetes.io/projected/08a59e43-43df-4b98-8824-2df9767fa85e-kube-api-access-66m4s\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:37:51.455969 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:51.455934    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"
Nov 02 18:37:51.456136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:51.455963    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf" event={"ID":"08a59e43-43df-4b98-8824-2df9767fa85e","Type":"ContainerDied","Data":"d46f419c493e08e467f1dc2fbb12e889047ea7ef223bf31091870fdab6e0459b"}
Nov 02 18:37:51.456136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:51.456036    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46f419c493e08e467f1dc2fbb12e889047ea7ef223bf31091870fdab6e0459b"
Nov 02 18:37:58.634661 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.634619    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"]
Nov 02 18:37:58.635130 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.635116    5042 cpu_manager.go:414] "containerMap: removing container" podUID="08a59e43-43df-4b98-8824-2df9767fa85e" containerName="revoke-keys"
Nov 02 18:37:58.635188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.635132    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a59e43-43df-4b98-8824-2df9767fa85e" containerName="revoke-keys"
Nov 02 18:37:58.635248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.635236    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a59e43-43df-4b98-8824-2df9767fa85e" containerName="revoke-keys"
Nov 02 18:37:58.638728 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.638710    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.643825 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.643798    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:37:58.644524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.643896    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:37:58.644524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.643945    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:37:58.644524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.644059    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:37:58.646428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.646394    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"]
Nov 02 18:37:58.790447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.790398    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5l7d\" (UniqueName: \"kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.790645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.790593    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.790726 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.790646    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.892468 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.892367    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5l7d\" (UniqueName: \"kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.892652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.892477    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.892652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.892521    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.893276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.893248    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.895260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.895239    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.900297 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.900276    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5l7d\" (UniqueName: \"kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d\") pod \"maas-api-revoke-sub-6b19338f-n2n2b\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:58.956193 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:58.956157    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:37:59.237761 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:59.237735    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"]
Nov 02 18:37:59.240174 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:37:59.240142    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2fcc32_debb_4d58_84a2_dcebb29c0930.slice/crio-af6344cbbd3ef959f5de112058a62033a887ea3826e4ffc8becbcf439d2f2109 WatchSource:0}: Error finding container af6344cbbd3ef959f5de112058a62033a887ea3826e4ffc8becbcf439d2f2109: Status 404 returned error can't find the container with id af6344cbbd3ef959f5de112058a62033a887ea3826e4ffc8becbcf439d2f2109
Nov 02 18:37:59.510439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:59.510402    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b" event={"ID":"fa2fcc32-debb-4d58-84a2-dcebb29c0930","Type":"ContainerStarted","Data":"6adb9fc9238cc33a31fd0fb98316a9acc78ab8560fcb6552ab3d5668a8cea39e"}
Nov 02 18:37:59.510633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:59.510469    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b" event={"ID":"fa2fcc32-debb-4d58-84a2-dcebb29c0930","Type":"ContainerStarted","Data":"af6344cbbd3ef959f5de112058a62033a887ea3826e4ffc8becbcf439d2f2109"}
Nov 02 18:37:59.528951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:37:59.528895    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b" podStartSLOduration=1.5288812360000001 podStartE2EDuration="1.528881236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:37:59.525472323 +0000 UTC m=+3892.104881709" watchObservedRunningTime="2026-11-02 18:37:59.528881236 +0000 UTC m=+3892.108290621"
Nov 02 18:38:00.518210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:00.518167    5042 generic.go:356] "Generic (PLEG): container finished" podID="fa2fcc32-debb-4d58-84a2-dcebb29c0930" containerID="6adb9fc9238cc33a31fd0fb98316a9acc78ab8560fcb6552ab3d5668a8cea39e" exitCode=0
Nov 02 18:38:00.518682 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:00.518211    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b" event={"ID":"fa2fcc32-debb-4d58-84a2-dcebb29c0930","Type":"ContainerDied","Data":"6adb9fc9238cc33a31fd0fb98316a9acc78ab8560fcb6552ab3d5668a8cea39e"}
Nov 02 18:38:01.667963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.667939    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:38:01.722333 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.722287    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca\") pod \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") "
Nov 02 18:38:01.722547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.722386    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d\" (UniqueName: \"kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d\") pod \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") "
Nov 02 18:38:01.722547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.722528    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca\") pod \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\" (UID: \"fa2fcc32-debb-4d58-84a2-dcebb29c0930\") "
Nov 02 18:38:01.722986 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.722953    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca" pod "fa2fcc32-debb-4d58-84a2-dcebb29c0930" (UID: "fa2fcc32-debb-4d58-84a2-dcebb29c0930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:38:01.724625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.724597    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca" pod "fa2fcc32-debb-4d58-84a2-dcebb29c0930" (UID: "fa2fcc32-debb-4d58-84a2-dcebb29c0930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:38:01.724730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.724699    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d" pod "fa2fcc32-debb-4d58-84a2-dcebb29c0930" (UID: "fa2fcc32-debb-4d58-84a2-dcebb29c0930"). InnerVolumeSpecName "kube-api-access-r5l7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:38:01.823361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.823279    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:38:01.823361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.823311    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa2fcc32-debb-4d58-84a2-dcebb29c0930-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:38:01.823361 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:01.823321    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5l7d\" (UniqueName: \"kubernetes.io/projected/fa2fcc32-debb-4d58-84a2-dcebb29c0930-kube-api-access-r5l7d\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:38:02.023344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:02.023304    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"]
Nov 02 18:38:02.026203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:02.026169    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b25bb23-qtvvp"]
Nov 02 18:38:02.532793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:02.532755    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b" event={"ID":"fa2fcc32-debb-4d58-84a2-dcebb29c0930","Type":"ContainerDied","Data":"af6344cbbd3ef959f5de112058a62033a887ea3826e4ffc8becbcf439d2f2109"}
Nov 02 18:38:02.532793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:02.532789    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"
Nov 02 18:38:02.533098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:02.532834    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6344cbbd3ef959f5de112058a62033a887ea3826e4ffc8becbcf439d2f2109"
Nov 02 18:38:03.542539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:03.542502    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e521b386-8648-4cd7-94e6-e1ef84a7ac84" path="/var/lib/kubelet/pods/e521b386-8648-4cd7-94e6-e1ef84a7ac84/volumes"
Nov 02 18:38:09.908689 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:09.908491    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:38:09.912084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:09.912051    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:38:09.912346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:09.912203    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:38:09.916207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:09.916190    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:38:10.664994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.664956    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"]
Nov 02 18:38:10.665906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.665809    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fa2fcc32-debb-4d58-84a2-dcebb29c0930" containerName="revoke-keys"
Nov 02 18:38:10.665906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.665854    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2fcc32-debb-4d58-84a2-dcebb29c0930" containerName="revoke-keys"
Nov 02 18:38:10.666086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.665987    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2fcc32-debb-4d58-84a2-dcebb29c0930" containerName="revoke-keys"
Nov 02 18:38:10.671093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.671069    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:10.720369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.720258    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs\") pod \"e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:10.720696 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.720669    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-744np\" (UniqueName: \"kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np\") pod \"e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:10.721241 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.721216    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-3c449aaf-kserve-self-signed-certs\""
Nov 02 18:38:10.721496 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.721479    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-3c449aaf-epp-sa-dockercfg-pwsbf\""
Nov 02 18:38:10.723679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.722994    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"]
Nov 02 18:38:10.829865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.826660    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs\") pod \"e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:10.829865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.826737    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-744np\" (UniqueName: \"kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np\") pod \"e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:10.862923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.862802    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs\") pod \"e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:10.869655 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:10.869567    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-744np\" (UniqueName: \"kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np\") pod \"e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") " pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:11.169466 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:11.169431    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:11.683806 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:11.683705    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"]
Nov 02 18:38:11.692789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:11.689811    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:38:12.026833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:12.026796    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"]
Nov 02 18:38:12.030772 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:12.030743    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3375b230-sf592"]
Nov 02 18:38:12.696588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:12.696541    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" event={"ID":"fe238b4b-306b-418c-96d0-bdffac12cfce","Type":"ContainerStarted","Data":"2fe858ea899babaa8859982e063aae4722efd612304f6d20cf632b78d04c9d1b"}
Nov 02 18:38:13.547978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:13.547693    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9de7f4-d729-4a03-8456-113fe6d8e55a" path="/var/lib/kubelet/pods/4f9de7f4-d729-4a03-8456-113fe6d8e55a/volumes"
Nov 02 18:38:14.742782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:14.742738    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" event={"ID":"fe238b4b-306b-418c-96d0-bdffac12cfce","Type":"ContainerStarted","Data":"745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6"}
Nov 02 18:38:14.744395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:14.744370    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:14.770647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:14.770584    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podStartSLOduration=2.711857665 podStartE2EDuration="4.77056459s" totalImagesPullingTime="2.058706925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:38:14.768684865 +0000 UTC m=+3907.348094252" watchObservedRunningTime="2026-11-02 18:38:14.77056459 +0000 UTC m=+3907.349973978"
Nov 02 18:38:18.539095 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:38:18.539014    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:38:23.022560 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:23.022520    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"]
Nov 02 18:38:23.024927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:23.024895    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee3a495-tfkl6"]
Nov 02 18:38:23.542609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:23.542569    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6d802e-f256-4e44-801e-dc891654530c" path="/var/lib/kubelet/pods/bf6d802e-f256-4e44-801e-dc891654530c/volumes"
Nov 02 18:38:33.022350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.022313    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"]
Nov 02 18:38:33.027859 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.027809    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-955448eb-6lc8l"]
Nov 02 18:38:33.542814 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.542779    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39499861-cad8-4bfa-a126-8c16f0158d16" path="/var/lib/kubelet/pods/39499861-cad8-4bfa-a126-8c16f0158d16/volumes"
Nov 02 18:38:33.600208 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.600173    5042 scope.go:122] "RemoveContainer" containerID="2a9571945fd6019fc44d87e71de44bc5aa612169db5a22b058415cf9e68d4e14"
Nov 02 18:38:33.609829 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.609810    5042 scope.go:122] "RemoveContainer" containerID="f443d08b0cbd0f00467c5ff57e81039b869c15d25af152f32af0099c809dbd8f"
Nov 02 18:38:33.619001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.618971    5042 scope.go:122] "RemoveContainer" containerID="cac8dbe9fbef993a147011f707fd14619c6f2b9511d3b6cd22fc6bac924f5181"
Nov 02 18:38:33.629079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:33.629058    5042 scope.go:122] "RemoveContainer" containerID="bd007dc5f69135c604d7bbf964d415a28a5a418877f6fcfdb07e336a6583dc72"
Nov 02 18:38:42.021344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:42.021297    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"]
Nov 02 18:38:42.023661 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:42.023636    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76445f74-tx7zp"]
Nov 02 18:38:43.542970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:43.542936    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e3fd54-5771-4d8a-a62c-149f4d517300" path="/var/lib/kubelet/pods/a3e3fd54-5771-4d8a-a62c-149f4d517300/volumes"
Nov 02 18:38:46.769310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:38:46.769279    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:38:48.538701 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:38:48.538660    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:39:03.023455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:03.023414    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"]
Nov 02 18:39:03.026440 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:03.026414    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d6fcd93-fj4tf"]
Nov 02 18:39:03.542355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:03.542318    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6adb9307-bac8-46a8-8ff8-fa45185feba5" path="/var/lib/kubelet/pods/6adb9307-bac8-46a8-8ff8-fa45185feba5/volumes"
Nov 02 18:39:12.524186 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:12.524146    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"]
Nov 02 18:39:16.769220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:16.769175    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:39:18.364067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:18.364026    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:39:18.364507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:18.364121    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:39:18.536003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:18.535959    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:39:18.536375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:18.536338    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:39:18.638272 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:39:18.638156    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:39:26.768861 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:26.768811    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:39:27.559794 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:27.559755    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main" containerID="cri-o://745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6" gracePeriod=45
Nov 02 18:39:33.678520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:33.678488    5042 scope.go:122] "RemoveContainer" containerID="062f6a4aab7c0d81b17409003016f6805a3bc43e1b54547af53715e393e93618"
Nov 02 18:39:33.688346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:33.688322    5042 scope.go:122] "RemoveContainer" containerID="8b4089685a62a160efb785785d040030600ec9da97d8e0a80a5454024bcd1da7"
Nov 02 18:39:33.697919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:33.697896    5042 scope.go:122] "RemoveContainer" containerID="0d3273674b75f07ff8ae03b639dc2b97579be4f394ed7adedd314351a1111c8c"
Nov 02 18:39:33.707126 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:33.707102    5042 scope.go:122] "RemoveContainer" containerID="0c69f0b7ce0775e62772e9e50b1c24ffc9ba80e04884c92f0c8c5ee70961274c"
Nov 02 18:39:36.768687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:36.768649    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:39:36.769136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:36.768759    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:39:44.030079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:44.030044    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"]
Nov 02 18:39:44.035093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:44.035063    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8b867-pct8z"]
Nov 02 18:39:45.548882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:45.547745    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627c8adc-845f-431f-b9f5-695a18be41d9" path="/var/lib/kubelet/pods/627c8adc-845f-431f-b9f5-695a18be41d9/volumes"
Nov 02 18:39:46.769246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:46.769206    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:39:48.542331 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:39:48.542233    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:39:55.027710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:55.027673    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"]
Nov 02 18:39:55.030685 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:55.030652    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de953a5d-2xvr8"]
Nov 02 18:39:55.543791 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:55.543741    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4" path="/var/lib/kubelet/pods/7e13bc1a-1c9d-47d1-8054-6e0cd1fd54b4/volumes"
Nov 02 18:39:56.768188 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:56.768152    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:39:57.829608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.829537    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:39:57.873707 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.873663    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np\" (UniqueName: \"kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np\") pod \"fe238b4b-306b-418c-96d0-bdffac12cfce\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") "
Nov 02 18:39:57.873950 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.873721    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs\") pod \"fe238b4b-306b-418c-96d0-bdffac12cfce\" (UID: \"fe238b4b-306b-418c-96d0-bdffac12cfce\") "
Nov 02 18:39:57.876879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.876820    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np" pod "fe238b4b-306b-418c-96d0-bdffac12cfce" (UID: "fe238b4b-306b-418c-96d0-bdffac12cfce"). InnerVolumeSpecName "kube-api-access-744np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:39:57.877018 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.876956    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs" pod "fe238b4b-306b-418c-96d0-bdffac12cfce" (UID: "fe238b4b-306b-418c-96d0-bdffac12cfce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:39:57.975511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.975468    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-744np\" (UniqueName: \"kubernetes.io/projected/fe238b4b-306b-418c-96d0-bdffac12cfce-kube-api-access-744np\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:39:57.975511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:57.975505    5042 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe238b4b-306b-418c-96d0-bdffac12cfce-tls-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:39:58.478327 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.478289    5042 generic.go:356] "Generic (PLEG): container finished" podID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerID="745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6" exitCode=0
Nov 02 18:39:58.478542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.478351    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"
Nov 02 18:39:58.478542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.478353    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" event={"ID":"fe238b4b-306b-418c-96d0-bdffac12cfce","Type":"ContainerDied","Data":"745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6"}
Nov 02 18:39:58.478542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.478482    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc" event={"ID":"fe238b4b-306b-418c-96d0-bdffac12cfce","Type":"ContainerDied","Data":"2fe858ea899babaa8859982e063aae4722efd612304f6d20cf632b78d04c9d1b"}
Nov 02 18:39:58.478542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.478511    5042 scope.go:122] "RemoveContainer" containerID="745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6"
Nov 02 18:39:58.489554 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.489533    5042 scope.go:122] "RemoveContainer" containerID="745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6"
Nov 02 18:39:58.489878 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:39:58.489856    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6\": container with ID starting with 745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6 not found: ID does not exist" containerID="745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6"
Nov 02 18:39:58.489964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.489886    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6"} err="failed to get container status \"745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6\": rpc error: code = NotFound desc = could not find container \"745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6\": container with ID starting with 745b895c589f591d0777246d21f164c3d4b466e3671ae2e0dacdd3f64e6e3ad6 not found: ID does not exist"
Nov 02 18:39:58.501215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.501187    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"]
Nov 02 18:39:58.508784 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:58.508761    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-3c449aaf-kserve-router-scheduler-6f848d6b55-2nlzc"]
Nov 02 18:39:59.541898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:39:59.541865    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" path="/var/lib/kubelet/pods/fe238b4b-306b-418c-96d0-bdffac12cfce/volumes"
Nov 02 18:40:18.543694 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:40:18.543657    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:40:18.558200 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.558160    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"]
Nov 02 18:40:18.558637 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.558622    5042 cpu_manager.go:414] "containerMap: removing container" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main"
Nov 02 18:40:18.558687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.558640    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main"
Nov 02 18:40:18.558747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.558737    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe238b4b-306b-418c-96d0-bdffac12cfce" containerName="main"
Nov 02 18:40:18.562058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.562035    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.567983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.567747    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:40:18.567983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.567750    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:40:18.567983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.567824    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:40:18.567983 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.567947    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:40:18.570887 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.570819    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"]
Nov 02 18:40:18.695060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.695006    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmwtd\" (UniqueName: \"kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.695295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.695210    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.695295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.695278    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.796751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.796641    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.796751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.796727    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.797075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.796793    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmwtd\" (UniqueName: \"kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.797385 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.797353    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.799481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.799460    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.805131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.805103    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmwtd\" (UniqueName: \"kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd\") pod \"maas-api-revoke-sub-439eb6ca-s54sw\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:18.881348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:18.881313    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:19.168064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:19.168035    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"]
Nov 02 18:40:19.170148 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:40:19.170122    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8200b351_eac1_4fa0_9f1e_d8a2171d4b3a.slice/crio-0bde4b9a54a4293dbb701455a08d3ebfba999c55229c88d959a4424c39f47e2e WatchSource:0}: Error finding container 0bde4b9a54a4293dbb701455a08d3ebfba999c55229c88d959a4424c39f47e2e: Status 404 returned error can't find the container with id 0bde4b9a54a4293dbb701455a08d3ebfba999c55229c88d959a4424c39f47e2e
Nov 02 18:40:19.626863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:19.626440    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw" event={"ID":"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a","Type":"ContainerStarted","Data":"0576bb4c525a49639de375ab4a132580feb872f2e093c20ecf19b10b6c141871"}
Nov 02 18:40:19.626863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:19.626505    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw" event={"ID":"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a","Type":"ContainerStarted","Data":"0bde4b9a54a4293dbb701455a08d3ebfba999c55229c88d959a4424c39f47e2e"}
Nov 02 18:40:19.662864 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:19.662015    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw" podStartSLOduration=1.66199379 podStartE2EDuration="1.66199379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:40:19.657230764 +0000 UTC m=+4032.236640151" watchObservedRunningTime="2026-11-02 18:40:19.66199379 +0000 UTC m=+4032.241403177"
Nov 02 18:40:20.634471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:20.634432    5042 generic.go:356] "Generic (PLEG): container finished" podID="8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" containerID="0576bb4c525a49639de375ab4a132580feb872f2e093c20ecf19b10b6c141871" exitCode=0
Nov 02 18:40:20.634930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:20.634523    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw" event={"ID":"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a","Type":"ContainerDied","Data":"0576bb4c525a49639de375ab4a132580feb872f2e093c20ecf19b10b6c141871"}
Nov 02 18:40:21.795394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.795367    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:21.842876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.833375    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd\" (UniqueName: \"kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd\") pod \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") "
Nov 02 18:40:21.842876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.833434    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca\") pod \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") "
Nov 02 18:40:21.842876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.833504    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca\") pod \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\" (UID: \"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a\") "
Nov 02 18:40:21.842876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.833918    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca" pod "8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" (UID: "8200b351-eac1-4fa0-9f1e-d8a2171d4b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:40:21.842876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.835866    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca" pod "8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" (UID: "8200b351-eac1-4fa0-9f1e-d8a2171d4b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:40:21.842876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.836126    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd" pod "8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" (UID: "8200b351-eac1-4fa0-9f1e-d8a2171d4b3a"). InnerVolumeSpecName "kube-api-access-dmwtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:40:21.934955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.934831    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:40:21.934955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.934898    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmwtd\" (UniqueName: \"kubernetes.io/projected/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-kube-api-access-dmwtd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:40:21.934955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:21.934913    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:40:22.647508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:22.647476    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"
Nov 02 18:40:22.647728 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:22.647511    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw" event={"ID":"8200b351-eac1-4fa0-9f1e-d8a2171d4b3a","Type":"ContainerDied","Data":"0bde4b9a54a4293dbb701455a08d3ebfba999c55229c88d959a4424c39f47e2e"}
Nov 02 18:40:22.647728 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:22.647560    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bde4b9a54a4293dbb701455a08d3ebfba999c55229c88d959a4424c39f47e2e"
Nov 02 18:40:33.747330 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:33.747299    5042 scope.go:122] "RemoveContainer" containerID="2834c53b27f9309a25f842963c0ec7e7120a4641069be51d12b1324645c0d500"
Nov 02 18:40:33.758324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:40:33.758299    5042 scope.go:122] "RemoveContainer" containerID="54bf3da646ac2e447c7c6dde0419b43fafe82c207b7745fdc685107f623496c0"
Nov 02 18:40:48.538380 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:40:48.538344    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:41:15.688736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.688688    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"]
Nov 02 18:41:15.689220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.689200    5042 cpu_manager.go:414] "containerMap: removing container" podUID="8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" containerName="revoke-keys"
Nov 02 18:41:15.689220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.689214    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" containerName="revoke-keys"
Nov 02 18:41:15.689307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.689296    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" containerName="revoke-keys"
Nov 02 18:41:15.692463 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.692445    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.697421 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.697381    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:41:15.697687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.697587    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:41:15.697687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.697650    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:41:15.697874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.697792    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:41:15.699719 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.699693    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"]
Nov 02 18:41:15.790032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.789997    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvg8p\" (UniqueName: \"kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.790236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.790066    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.790236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.790154    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.891435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.891403    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvg8p\" (UniqueName: \"kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.891633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.891460    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.891633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.891492    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.892225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.892191    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.894311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.894292    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:15.900757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:15.900727    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvg8p\" (UniqueName: \"kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:16.009255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:16.009146    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:16.285292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:16.285262    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"]
Nov 02 18:41:16.287518 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:41:16.287479    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b9031e_3920_41f7_98f2_4e3eb13be245.slice/crio-a32579ab5bdcec5db7bbb5058f63cc5aba0d6f5a137b01aa1c236c2707731c5d WatchSource:0}: Error finding container a32579ab5bdcec5db7bbb5058f63cc5aba0d6f5a137b01aa1c236c2707731c5d: Status 404 returned error can't find the container with id a32579ab5bdcec5db7bbb5058f63cc5aba0d6f5a137b01aa1c236c2707731c5d
Nov 02 18:41:16.971094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:16.971056    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz" event={"ID":"d6b9031e-3920-41f7-98f2-4e3eb13be245","Type":"ContainerStarted","Data":"8d0c992604d255eb981fe5b8ab311bae6bb33fc6acbf6af334c4dbd86f0e2985"}
Nov 02 18:41:16.971546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:16.971114    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz" event={"ID":"d6b9031e-3920-41f7-98f2-4e3eb13be245","Type":"ContainerStarted","Data":"a32579ab5bdcec5db7bbb5058f63cc5aba0d6f5a137b01aa1c236c2707731c5d"}
Nov 02 18:41:16.988398 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:16.988349    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz" podStartSLOduration=1.988335995 podStartE2EDuration="1.988335995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:16.986308769 +0000 UTC m=+4089.565718159" watchObservedRunningTime="2026-11-02 18:41:16.988335995 +0000 UTC m=+4089.567745417"
Nov 02 18:41:17.978614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:17.978534    5042 generic.go:356] "Generic (PLEG): container finished" podID="d6b9031e-3920-41f7-98f2-4e3eb13be245" containerID="8d0c992604d255eb981fe5b8ab311bae6bb33fc6acbf6af334c4dbd86f0e2985" exitCode=0
Nov 02 18:41:17.979063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:17.978622    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz" event={"ID":"d6b9031e-3920-41f7-98f2-4e3eb13be245","Type":"ContainerDied","Data":"8d0c992604d255eb981fe5b8ab311bae6bb33fc6acbf6af334c4dbd86f0e2985"}
Nov 02 18:41:18.544475 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:41:18.544438    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:41:19.143474 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.143450    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:19.227153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.227118    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca\") pod \"d6b9031e-3920-41f7-98f2-4e3eb13be245\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") "
Nov 02 18:41:19.227351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.227263    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p\" (UniqueName: \"kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p\") pod \"d6b9031e-3920-41f7-98f2-4e3eb13be245\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") "
Nov 02 18:41:19.227351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.227284    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca\") pod \"d6b9031e-3920-41f7-98f2-4e3eb13be245\" (UID: \"d6b9031e-3920-41f7-98f2-4e3eb13be245\") "
Nov 02 18:41:19.227587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.227558    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca" pod "d6b9031e-3920-41f7-98f2-4e3eb13be245" (UID: "d6b9031e-3920-41f7-98f2-4e3eb13be245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:41:19.229577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.229546    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca" pod "d6b9031e-3920-41f7-98f2-4e3eb13be245" (UID: "d6b9031e-3920-41f7-98f2-4e3eb13be245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:41:19.229577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.229556    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p" pod "d6b9031e-3920-41f7-98f2-4e3eb13be245" (UID: "d6b9031e-3920-41f7-98f2-4e3eb13be245"). InnerVolumeSpecName "kube-api-access-jvg8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:41:19.328501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.328462    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvg8p\" (UniqueName: \"kubernetes.io/projected/d6b9031e-3920-41f7-98f2-4e3eb13be245-kube-api-access-jvg8p\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:41:19.328501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.328494    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:41:19.328501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.328507    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b9031e-3920-41f7-98f2-4e3eb13be245-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:41:19.992824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.992792    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"
Nov 02 18:41:19.993049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.992845    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz" event={"ID":"d6b9031e-3920-41f7-98f2-4e3eb13be245","Type":"ContainerDied","Data":"a32579ab5bdcec5db7bbb5058f63cc5aba0d6f5a137b01aa1c236c2707731c5d"}
Nov 02 18:41:19.993049 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:19.992904    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32579ab5bdcec5db7bbb5058f63cc5aba0d6f5a137b01aa1c236c2707731c5d"
Nov 02 18:41:27.108968 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:27.108910    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"]
Nov 02 18:41:27.112757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:27.112725    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c3df6f-h9mwz"]
Nov 02 18:41:27.549328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:27.549286    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b9031e-3920-41f7-98f2-4e3eb13be245" path="/var/lib/kubelet/pods/d6b9031e-3920-41f7-98f2-4e3eb13be245/volumes"
Nov 02 18:41:33.797301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:33.797259    5042 scope.go:122] "RemoveContainer" containerID="907d5a5b8c6ed4c1239e59bfee8f4e6db506a131f2680de0f4c07b196b6f0fed"
Nov 02 18:41:33.817911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:33.817879    5042 scope.go:122] "RemoveContainer" containerID="7ee5123d2603483927b95f95fa40f61c38351326b80d0da6038abfd4aa725464"
Nov 02 18:41:35.029876 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:35.026216    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"]
Nov 02 18:41:35.039869 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:35.038562    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-ctbbt"]
Nov 02 18:41:35.555301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:41:35.555255    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127aaba2-9099-4be8-b192-4ded210aea33" path="/var/lib/kubelet/pods/127aaba2-9099-4be8-b192-4ded210aea33/volumes"
Nov 02 18:41:48.539095 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:41:48.539049    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:42:04.973266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.973225    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"]
Nov 02 18:42:04.973730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.973693    5042 cpu_manager.go:414] "containerMap: removing container" podUID="d6b9031e-3920-41f7-98f2-4e3eb13be245" containerName="revoke-keys"
Nov 02 18:42:04.973730 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.973705    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b9031e-3920-41f7-98f2-4e3eb13be245" containerName="revoke-keys"
Nov 02 18:42:04.973808 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.973776    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b9031e-3920-41f7-98f2-4e3eb13be245" containerName="revoke-keys"
Nov 02 18:42:04.977253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.977231    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:04.982410 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.982386    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:42:04.982580 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.982559    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:42:04.982674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.982522    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:42:04.982674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.982398    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:42:04.984586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.984562    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"]
Nov 02 18:42:04.986815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.986793    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:04.986948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.986827    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:04.986948 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:04.986884    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxghl\" (UniqueName: \"kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.088518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.088462    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxghl\" (UniqueName: \"kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.088752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.088643    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.088752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.088674    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.089577 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.089506    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.091326 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.091305    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.096968 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.096944    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxghl\" (UniqueName: \"kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl\") pod \"maas-api-revoke-sub-452b6d7b-hp6mt\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.295235 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.295137    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:05.574407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:05.574319    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"]
Nov 02 18:42:05.578325 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:42:05.578278    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b477d1_e4b3_4f5b_99f7_d7badc83c63d.slice/crio-e89dfcaa7391589d90884694391f618ea82b0afc2778f1e206a2295adce99714 WatchSource:0}: Error finding container e89dfcaa7391589d90884694391f618ea82b0afc2778f1e206a2295adce99714: Status 404 returned error can't find the container with id e89dfcaa7391589d90884694391f618ea82b0afc2778f1e206a2295adce99714
Nov 02 18:42:06.330729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:06.330627    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt" event={"ID":"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d","Type":"ContainerStarted","Data":"29b3d05741748637b5c7138a93347b4e69ee42e397610b34a198203e140398e4"}
Nov 02 18:42:06.330729 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:06.330695    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt" event={"ID":"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d","Type":"ContainerStarted","Data":"e89dfcaa7391589d90884694391f618ea82b0afc2778f1e206a2295adce99714"}
Nov 02 18:42:06.348905 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:06.348813    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt" podStartSLOduration=2.348798757 podStartE2EDuration="2.348798757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:42:06.347131174 +0000 UTC m=+4138.926540561" watchObservedRunningTime="2026-11-02 18:42:06.348798757 +0000 UTC m=+4138.928208143"
Nov 02 18:42:07.358860 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:07.358802    5042 generic.go:356] "Generic (PLEG): container finished" podID="c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" containerID="29b3d05741748637b5c7138a93347b4e69ee42e397610b34a198203e140398e4" exitCode=0
Nov 02 18:42:07.359407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:07.358918    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt" event={"ID":"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d","Type":"ContainerDied","Data":"29b3d05741748637b5c7138a93347b4e69ee42e397610b34a198203e140398e4"}
Nov 02 18:42:08.647662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:08.647183    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:42:08.647662 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:08.647258    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:42:08.985959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:08.980199    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:09.114453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.114400    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca\") pod \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") "
Nov 02 18:42:09.115090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.115058    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl\" (UniqueName: \"kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl\") pod \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") "
Nov 02 18:42:09.116287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.116260    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca\") pod \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\" (UID: \"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d\") "
Nov 02 18:42:09.121663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.121070    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl" pod "c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" (UID: "c3b477d1-e4b3-4f5b-99f7-d7badc83c63d"). InnerVolumeSpecName "kube-api-access-dxghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:42:09.128712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.128677    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca" pod "c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" (UID: "c3b477d1-e4b3-4f5b-99f7-d7badc83c63d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:42:09.133404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.133162    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca" pod "c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" (UID: "c3b477d1-e4b3-4f5b-99f7-d7badc83c63d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:42:09.217738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.217700    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:09.217738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.217741    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxghl\" (UniqueName: \"kubernetes.io/projected/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-kube-api-access-dxghl\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:09.218022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.217760    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:09.372751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.372655    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"
Nov 02 18:42:09.372751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.372688    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt" event={"ID":"c3b477d1-e4b3-4f5b-99f7-d7badc83c63d","Type":"ContainerDied","Data":"e89dfcaa7391589d90884694391f618ea82b0afc2778f1e206a2295adce99714"}
Nov 02 18:42:09.373004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:09.372765    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89dfcaa7391589d90884694391f618ea82b0afc2778f1e206a2295adce99714"
Nov 02 18:42:14.026543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:14.026501    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"]
Nov 02 18:42:14.029106 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:14.029075    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-h69n4"]
Nov 02 18:42:15.543661 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:15.543628    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4fc0a9-8e92-4db4-9150-79f3047eb1dd" path="/var/lib/kubelet/pods/dc4fc0a9-8e92-4db4-9150-79f3047eb1dd/volumes"
Nov 02 18:42:18.549157 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:42:18.549113    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:42:18.638467 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.638428    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"]
Nov 02 18:42:18.639154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.639126    5042 cpu_manager.go:414] "containerMap: removing container" podUID="c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" containerName="revoke-keys"
Nov 02 18:42:18.639276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.639155    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" containerName="revoke-keys"
Nov 02 18:42:18.639340 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.639324    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" containerName="revoke-keys"
Nov 02 18:42:18.643888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.643580    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.648500 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.648472    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"]
Nov 02 18:42:18.650108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.650082    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:42:18.650693 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.650673    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:42:18.652221 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.652196    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:42:18.652351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.652307    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:42:18.727519 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.727479    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.727747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.727551    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbtf\" (UniqueName: \"kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.727747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.727601    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.832916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.831391    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.832916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.831560    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.832916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.831610    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbtf\" (UniqueName: \"kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.832916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.832788    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.856891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.856858    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbtf\" (UniqueName: \"kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.867087 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.867038    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-2d9x7\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:18.984868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:18.976028    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:19.452383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:19.452088    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"]
Nov 02 18:42:20.521489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:20.521446    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7" event={"ID":"ef4a0aba-7029-4d01-afef-a78c0ea17c56","Type":"ContainerStarted","Data":"ff4026e3a543bef4be977ca21ac76a5fae6503d2107888452453cdf29a0ec812"}
Nov 02 18:42:20.522128 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:20.522099    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7" event={"ID":"ef4a0aba-7029-4d01-afef-a78c0ea17c56","Type":"ContainerStarted","Data":"ed32029d1cf4eb39d0ef6a413ce6d04454ee3df5c611f322443efbf5e956ca38"}
Nov 02 18:42:20.550891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:20.550780    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7" podStartSLOduration=2.5507443480000003 podStartE2EDuration="2.550744348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:42:20.543301798 +0000 UTC m=+4153.122711188" watchObservedRunningTime="2026-11-02 18:42:20.550744348 +0000 UTC m=+4153.130153735"
Nov 02 18:42:21.530504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:21.530462    5042 generic.go:356] "Generic (PLEG): container finished" podID="ef4a0aba-7029-4d01-afef-a78c0ea17c56" containerID="ff4026e3a543bef4be977ca21ac76a5fae6503d2107888452453cdf29a0ec812" exitCode=0
Nov 02 18:42:21.531085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:21.530625    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7" event={"ID":"ef4a0aba-7029-4d01-afef-a78c0ea17c56","Type":"ContainerDied","Data":"ff4026e3a543bef4be977ca21ac76a5fae6503d2107888452453cdf29a0ec812"}
Nov 02 18:42:23.090115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.090073    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:23.232868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.211018    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca\") pod \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") "
Nov 02 18:42:23.232868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.211150    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca\") pod \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") "
Nov 02 18:42:23.232868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.211191    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf\" (UniqueName: \"kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf\") pod \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\" (UID: \"ef4a0aba-7029-4d01-afef-a78c0ea17c56\") "
Nov 02 18:42:23.244955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.243093    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca" pod "ef4a0aba-7029-4d01-afef-a78c0ea17c56" (UID: "ef4a0aba-7029-4d01-afef-a78c0ea17c56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:42:23.262318 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.262259    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf" pod "ef4a0aba-7029-4d01-afef-a78c0ea17c56" (UID: "ef4a0aba-7029-4d01-afef-a78c0ea17c56"). InnerVolumeSpecName "kube-api-access-rtbtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:42:23.326395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.313218    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:23.326395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.313263    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtbtf\" (UniqueName: \"kubernetes.io/projected/ef4a0aba-7029-4d01-afef-a78c0ea17c56-kube-api-access-rtbtf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:23.348057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.347986    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca" pod "ef4a0aba-7029-4d01-afef-a78c0ea17c56" (UID: "ef4a0aba-7029-4d01-afef-a78c0ea17c56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:42:23.430866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.427166    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef4a0aba-7029-4d01-afef-a78c0ea17c56-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:23.551004 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.550904    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7" event={"ID":"ef4a0aba-7029-4d01-afef-a78c0ea17c56","Type":"ContainerDied","Data":"ed32029d1cf4eb39d0ef6a413ce6d04454ee3df5c611f322443efbf5e956ca38"}
Nov 02 18:42:23.551277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.551249    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed32029d1cf4eb39d0ef6a413ce6d04454ee3df5c611f322443efbf5e956ca38"
Nov 02 18:42:23.551277 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.550924    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"
Nov 02 18:42:23.603520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.603186    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:42:23.603520 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:23.603257    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:42:33.871771 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:33.871730    5042 scope.go:122] "RemoveContainer" containerID="f0b4039bb0d8c0927314c0f00262886d39f41e5286e6a757c9acea6c717017fb"
Nov 02 18:42:33.915169 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:33.915120    5042 scope.go:122] "RemoveContainer" containerID="5af4c57bddfed7f2f58b66547f27f75ee217f5f250722c19d2fce31bcf07cce2"
Nov 02 18:42:35.788015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.787977    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"]
Nov 02 18:42:35.788623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.788602    5042 cpu_manager.go:414] "containerMap: removing container" podUID="ef4a0aba-7029-4d01-afef-a78c0ea17c56" containerName="revoke-keys"
Nov 02 18:42:35.788734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.788626    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef4a0aba-7029-4d01-afef-a78c0ea17c56" containerName="revoke-keys"
Nov 02 18:42:35.788795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.788780    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef4a0aba-7029-4d01-afef-a78c0ea17c56" containerName="revoke-keys"
Nov 02 18:42:35.793439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.793415    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:35.799809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.798649    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:42:35.799809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.798804    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:42:35.799809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.798911    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:42:35.799809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.798948    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:42:35.799809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.799432    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"]
Nov 02 18:42:35.925824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.925783    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrptz\" (UniqueName: \"kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:35.926058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.925869    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:35.926058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:35.925960    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.022747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.022702    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"]
Nov 02 18:42:36.025002 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.024975    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wbd8m"]
Nov 02 18:42:36.026890 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.026833    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrptz\" (UniqueName: \"kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.027035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.026944    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.027035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.026976    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.027663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.027641    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.029760 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.029737    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.035022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.034997    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrptz\" (UniqueName: \"kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz\") pod \"maas-api-revoke-sub-13c08e66-24ml9\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.114022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.113931    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:36.394246 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.394206    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"]
Nov 02 18:42:36.395590 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:42:36.395556    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0a3daf_4171_45b6_95de_c97bfeb6ce08.slice/crio-23fe92b4a55dde8b1467fef3eed49e6084dcfa6ff4f8fd74c1e0b66b2b12a7a1 WatchSource:0}: Error finding container 23fe92b4a55dde8b1467fef3eed49e6084dcfa6ff4f8fd74c1e0b66b2b12a7a1: Status 404 returned error can't find the container with id 23fe92b4a55dde8b1467fef3eed49e6084dcfa6ff4f8fd74c1e0b66b2b12a7a1
Nov 02 18:42:36.686377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:36.686334    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9" event={"ID":"7e0a3daf-4171-45b6-95de-c97bfeb6ce08","Type":"ContainerStarted","Data":"23fe92b4a55dde8b1467fef3eed49e6084dcfa6ff4f8fd74c1e0b66b2b12a7a1"}
Nov 02 18:42:37.542271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:37.542183    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23285ac0-1b5d-47b8-98d5-5d1eaa3e2751" path="/var/lib/kubelet/pods/23285ac0-1b5d-47b8-98d5-5d1eaa3e2751/volumes"
Nov 02 18:42:37.694084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:37.694053    5042 generic.go:356] "Generic (PLEG): container finished" podID="7e0a3daf-4171-45b6-95de-c97bfeb6ce08" containerID="7445492808e4b6fbcbe1f3e60d6b04f98a241d807ee09fdbe0d521c134bdc428" exitCode=0
Nov 02 18:42:37.694274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:37.694144    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9" event={"ID":"7e0a3daf-4171-45b6-95de-c97bfeb6ce08","Type":"ContainerDied","Data":"7445492808e4b6fbcbe1f3e60d6b04f98a241d807ee09fdbe0d521c134bdc428"}
Nov 02 18:42:38.862984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.862951    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:38.956695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.956594    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca\") pod \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") "
Nov 02 18:42:38.956695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.956638    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz\" (UniqueName: \"kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz\") pod \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") "
Nov 02 18:42:38.956999 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.956754    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca\") pod \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\" (UID: \"7e0a3daf-4171-45b6-95de-c97bfeb6ce08\") "
Nov 02 18:42:38.957056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.957005    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca" pod "7e0a3daf-4171-45b6-95de-c97bfeb6ce08" (UID: "7e0a3daf-4171-45b6-95de-c97bfeb6ce08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:42:38.957180 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.957164    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:38.959022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.958998    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz" pod "7e0a3daf-4171-45b6-95de-c97bfeb6ce08" (UID: "7e0a3daf-4171-45b6-95de-c97bfeb6ce08"). InnerVolumeSpecName "kube-api-access-rrptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:42:38.959133 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:38.959000    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca" pod "7e0a3daf-4171-45b6-95de-c97bfeb6ce08" (UID: "7e0a3daf-4171-45b6-95de-c97bfeb6ce08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:42:39.058025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:39.057989    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrptz\" (UniqueName: \"kubernetes.io/projected/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-kube-api-access-rrptz\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:39.058025 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:39.058022    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e0a3daf-4171-45b6-95de-c97bfeb6ce08-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:42:39.707875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:39.707829    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"
Nov 02 18:42:39.707875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:39.707860    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9" event={"ID":"7e0a3daf-4171-45b6-95de-c97bfeb6ce08","Type":"ContainerDied","Data":"23fe92b4a55dde8b1467fef3eed49e6084dcfa6ff4f8fd74c1e0b66b2b12a7a1"}
Nov 02 18:42:39.708108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:39.707907    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23fe92b4a55dde8b1467fef3eed49e6084dcfa6ff4f8fd74c1e0b66b2b12a7a1"
Nov 02 18:42:48.550634 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:42:48.550542    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:42:51.024568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:51.024530    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"]
Nov 02 18:42:51.026976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:51.026953    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-s2kqf"]
Nov 02 18:42:51.542373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:42:51.542340    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a59e43-43df-4b98-8824-2df9767fa85e" path="/var/lib/kubelet/pods/08a59e43-43df-4b98-8824-2df9767fa85e/volumes"
Nov 02 18:43:02.023734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:02.023688    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"]
Nov 02 18:43:02.026013 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:02.025988    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n2n2b"]
Nov 02 18:43:03.543955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:03.543913    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2fcc32-debb-4d58-84a2-dcebb29c0930" path="/var/lib/kubelet/pods/fa2fcc32-debb-4d58-84a2-dcebb29c0930/volumes"
Nov 02 18:43:09.961895 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:09.961860    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:43:09.964573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:09.964551    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:43:09.964708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:09.964555    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:43:09.967454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:09.967435    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:43:18.541078 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:43:18.541040    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:43:34.072366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:34.072321    5042 scope.go:122] "RemoveContainer" containerID="b0285fb680f8b882ad7a414b3a3b6b606e6396200a6b387f99872af428c159c7"
Nov 02 18:43:34.085743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:43:34.085716    5042 scope.go:122] "RemoveContainer" containerID="86499bfb6332fd637ba26815ad567d623fd27c7fc9df33e47b184047a99134be"
Nov 02 18:43:48.538973 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:43:48.538934    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:44:18.538504 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:44:18.538420    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:44:34.143420 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:44:34.143387    5042 scope.go:122] "RemoveContainer" containerID="6adb9fc9238cc33a31fd0fb98316a9acc78ab8560fcb6552ab3d5668a8cea39e"
Nov 02 18:44:34.152768 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:44:34.152745    5042 scope.go:122] "RemoveContainer" containerID="99d9c091638918e405824e2950fa1b9e0e19c055fbbff6113ef743ba191ecab6"
Nov 02 18:44:48.538424 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:44:48.538379    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:45:00.141120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.141083    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"]
Nov 02 18:45:00.141594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.141550    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7e0a3daf-4171-45b6-95de-c97bfeb6ce08" containerName="revoke-keys"
Nov 02 18:45:00.141594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.141562    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0a3daf-4171-45b6-95de-c97bfeb6ce08" containerName="revoke-keys"
Nov 02 18:45:00.141681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.141626    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0a3daf-4171-45b6-95de-c97bfeb6ce08" containerName="revoke-keys"
Nov 02 18:45:00.144918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.144900    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.150064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.150022    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:45:00.150064 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.150022    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:45:00.150303 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.150095    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:45:00.153346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.152977    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"]
Nov 02 18:45:00.281944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.281912    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894085-wks4w\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.282158 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.281981    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhxh\" (UniqueName: \"kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh\") pod \"maas-api-key-cleanup-29894085-wks4w\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.383539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.383497    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894085-wks4w\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.383765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.383573    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhxh\" (UniqueName: \"kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh\") pod \"maas-api-key-cleanup-29894085-wks4w\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.384271 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.384242    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894085-wks4w\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.392775 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.392719    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhxh\" (UniqueName: \"kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh\") pod \"maas-api-key-cleanup-29894085-wks4w\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.460778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.460739    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:00.736809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:00.736781    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"]
Nov 02 18:45:00.739012 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:45:00.738976    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da1f5f5_19ce_46bc_972e_73732c8540a9.slice/crio-196185875a557de3d5ea95adf73f1343925290af91c9e5efb6aab43e26fb2104 WatchSource:0}: Error finding container 196185875a557de3d5ea95adf73f1343925290af91c9e5efb6aab43e26fb2104: Status 404 returned error can't find the container with id 196185875a557de3d5ea95adf73f1343925290af91c9e5efb6aab43e26fb2104
Nov 02 18:45:01.553006 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:01.552961    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" event={"ID":"6da1f5f5-19ce-46bc-972e-73732c8540a9","Type":"ContainerStarted","Data":"8dcf6832eb75f6cd1cf78e366c7359024dfb5d94c7cb8f1209f4accb7d5bb086"}
Nov 02 18:45:01.553543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:01.553028    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" event={"ID":"6da1f5f5-19ce-46bc-972e-73732c8540a9","Type":"ContainerStarted","Data":"196185875a557de3d5ea95adf73f1343925290af91c9e5efb6aab43e26fb2104"}
Nov 02 18:45:01.571588 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:01.571535    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" podStartSLOduration=1.571520588 podStartE2EDuration="1.571520588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:01.569604616 +0000 UTC m=+4314.149014006" watchObservedRunningTime="2026-11-02 18:45:01.571520588 +0000 UTC m=+4314.150929974"
Nov 02 18:45:02.559697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:02.559663    5042 generic.go:356] "Generic (PLEG): container finished" podID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerID="8dcf6832eb75f6cd1cf78e366c7359024dfb5d94c7cb8f1209f4accb7d5bb086" exitCode=60
Nov 02 18:45:02.560114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:02.559757    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" event={"ID":"6da1f5f5-19ce-46bc-972e-73732c8540a9","Type":"ContainerDied","Data":"8dcf6832eb75f6cd1cf78e366c7359024dfb5d94c7cb8f1209f4accb7d5bb086"}
Nov 02 18:45:02.560114 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:02.560030    5042 scope.go:122] "RemoveContainer" containerID="8dcf6832eb75f6cd1cf78e366c7359024dfb5d94c7cb8f1209f4accb7d5bb086"
Nov 02 18:45:03.573997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:03.573361    5042 generic.go:356] "Generic (PLEG): container finished" podID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerID="0a3cab868fec8ea766a7fcfbc53eaa4d122cd4b4ff7051459b68a91040311329" exitCode=60
Nov 02 18:45:03.573997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:03.573510    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" event={"ID":"6da1f5f5-19ce-46bc-972e-73732c8540a9","Type":"ContainerDied","Data":"0a3cab868fec8ea766a7fcfbc53eaa4d122cd4b4ff7051459b68a91040311329"}
Nov 02 18:45:03.573997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:03.573613    5042 scope.go:122] "RemoveContainer" containerID="8dcf6832eb75f6cd1cf78e366c7359024dfb5d94c7cb8f1209f4accb7d5bb086"
Nov 02 18:45:03.573997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:03.573922    5042 scope.go:122] "RemoveContainer" containerID="0a3cab868fec8ea766a7fcfbc53eaa4d122cd4b4ff7051459b68a91040311329"
Nov 02 18:45:03.574601 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:45:03.574242    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894085-wks4w_odh-ai-gateway-infra(6da1f5f5-19ce-46bc-972e-73732c8540a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9"
Nov 02 18:45:04.582632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:04.582605    5042 scope.go:122] "RemoveContainer" containerID="0a3cab868fec8ea766a7fcfbc53eaa4d122cd4b4ff7051459b68a91040311329"
Nov 02 18:45:04.583090 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:45:04.582891    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894085-wks4w_odh-ai-gateway-infra(6da1f5f5-19ce-46bc-972e-73732c8540a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9"
Nov 02 18:45:18.535698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:18.535666    5042 scope.go:122] "RemoveContainer" containerID="0a3cab868fec8ea766a7fcfbc53eaa4d122cd4b4ff7051459b68a91040311329"
Nov 02 18:45:18.539538 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:45:18.539507    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:45:19.666931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:19.666825    5042 generic.go:356] "Generic (PLEG): container finished" podID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerID="ac02d61df093f0bdd01d80c85ddf36701eaabb96f21e6e8e92b5abc02917e048" exitCode=60
Nov 02 18:45:19.666931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:19.666902    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" event={"ID":"6da1f5f5-19ce-46bc-972e-73732c8540a9","Type":"ContainerDied","Data":"ac02d61df093f0bdd01d80c85ddf36701eaabb96f21e6e8e92b5abc02917e048"}
Nov 02 18:45:19.667447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:19.666969    5042 scope.go:122] "RemoveContainer" containerID="0a3cab868fec8ea766a7fcfbc53eaa4d122cd4b4ff7051459b68a91040311329"
Nov 02 18:45:19.667447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:19.667319    5042 scope.go:122] "RemoveContainer" containerID="ac02d61df093f0bdd01d80c85ddf36701eaabb96f21e6e8e92b5abc02917e048"
Nov 02 18:45:19.667600 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:45:19.667574    5042 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894085-wks4w_odh-ai-gateway-infra(6da1f5f5-19ce-46bc-972e-73732c8540a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9"
Nov 02 18:45:20.691723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:20.691686    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"]
Nov 02 18:45:20.853072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:20.853046    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:20.900228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:20.900184    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca\") pod \"6da1f5f5-19ce-46bc-972e-73732c8540a9\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") "
Nov 02 18:45:20.900444 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:20.900293    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh\" (UniqueName: \"kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh\") pod \"6da1f5f5-19ce-46bc-972e-73732c8540a9\" (UID: \"6da1f5f5-19ce-46bc-972e-73732c8540a9\") "
Nov 02 18:45:20.900666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:20.900635    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca" pod "6da1f5f5-19ce-46bc-972e-73732c8540a9" (UID: "6da1f5f5-19ce-46bc-972e-73732c8540a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:20.902769 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:20.902740    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh" pod "6da1f5f5-19ce-46bc-972e-73732c8540a9" (UID: "6da1f5f5-19ce-46bc-972e-73732c8540a9"). InnerVolumeSpecName "kube-api-access-jzhxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:21.001335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.001249    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6da1f5f5-19ce-46bc-972e-73732c8540a9-maas-api-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:21.001335 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.001278    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhxh\" (UniqueName: \"kubernetes.io/projected/6da1f5f5-19ce-46bc-972e-73732c8540a9-kube-api-access-jzhxh\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:21.681248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.681207    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w" event={"ID":"6da1f5f5-19ce-46bc-972e-73732c8540a9","Type":"ContainerDied","Data":"196185875a557de3d5ea95adf73f1343925290af91c9e5efb6aab43e26fb2104"}
Nov 02 18:45:21.681248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.681240    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"
Nov 02 18:45:21.681506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.681284    5042 scope.go:122] "RemoveContainer" containerID="ac02d61df093f0bdd01d80c85ddf36701eaabb96f21e6e8e92b5abc02917e048"
Nov 02 18:45:21.698828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.698797    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"]
Nov 02 18:45:21.704636 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:21.704603    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-wks4w"]
Nov 02 18:45:22.022931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:22.022879    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"]
Nov 02 18:45:22.025082 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:22.025060    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-439eb6ca-s54sw"]
Nov 02 18:45:23.543154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:23.543113    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" path="/var/lib/kubelet/pods/6da1f5f5-19ce-46bc-972e-73732c8540a9/volumes"
Nov 02 18:45:23.543586 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:23.543549    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8200b351-eac1-4fa0-9f1e-d8a2171d4b3a" path="/var/lib/kubelet/pods/8200b351-eac1-4fa0-9f1e-d8a2171d4b3a/volumes"
Nov 02 18:45:32.885407 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885362    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"]
Nov 02 18:45:32.885946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885814    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.885946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885825    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.885946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885871    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.885946 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885877    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.886097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885952    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.886097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885964    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.886097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.885970    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:32.890682 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.890659    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:32.896001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.895975    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:45:32.896187 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.895999    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:45:32.896269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.896033    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:45:32.896269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.896072    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:45:32.896734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.896709    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"]
Nov 02 18:45:32.906955 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.906917    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:32.907130 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.907008    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftjt\" (UniqueName: \"kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:32.907130 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:32.907059    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.007623 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.007575    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.007893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.007635    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftjt\" (UniqueName: \"kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.007893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.007663    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.008377 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.008349    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.010288 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.010268    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.017710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.017683    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftjt\" (UniqueName: \"kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt\") pod \"maas-api-revoke-sub-c1e271a9-vtljg\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.207625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.207584    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:33.484609 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.484578    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"]
Nov 02 18:45:33.487341 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:45:33.487306    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f0d812_72f4_488d_8487_72037df581eb.slice/crio-908093d00c412e559243e0a2095d35d42b83591cecdcfb95f44e67cc821146ef WatchSource:0}: Error finding container 908093d00c412e559243e0a2095d35d42b83591cecdcfb95f44e67cc821146ef: Status 404 returned error can't find the container with id 908093d00c412e559243e0a2095d35d42b83591cecdcfb95f44e67cc821146ef
Nov 02 18:45:33.832778 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.832736    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg" event={"ID":"31f0d812-72f4-488d-8487-72037df581eb","Type":"ContainerStarted","Data":"72942d1e5e7d7f9d17a4b8a7ae9b8015ba0772e364fd77c3805fe89a828b607b"}
Nov 02 18:45:33.832996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.832802    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg" event={"ID":"31f0d812-72f4-488d-8487-72037df581eb","Type":"ContainerStarted","Data":"908093d00c412e559243e0a2095d35d42b83591cecdcfb95f44e67cc821146ef"}
Nov 02 18:45:33.849710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:33.849655    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg" podStartSLOduration=1.8496422209999999 podStartE2EDuration="1.849642221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:33.847643405 +0000 UTC m=+4346.427052791" watchObservedRunningTime="2026-11-02 18:45:33.849642221 +0000 UTC m=+4346.429051607"
Nov 02 18:45:34.840543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:34.840510    5042 generic.go:356] "Generic (PLEG): container finished" podID="31f0d812-72f4-488d-8487-72037df581eb" containerID="72942d1e5e7d7f9d17a4b8a7ae9b8015ba0772e364fd77c3805fe89a828b607b" exitCode=0
Nov 02 18:45:34.841066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:34.840599    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg" event={"ID":"31f0d812-72f4-488d-8487-72037df581eb","Type":"ContainerDied","Data":"72942d1e5e7d7f9d17a4b8a7ae9b8015ba0772e364fd77c3805fe89a828b607b"}
Nov 02 18:45:35.986249 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:35.986222    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:36.038575 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.038525    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca\") pod \"31f0d812-72f4-488d-8487-72037df581eb\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") "
Nov 02 18:45:36.038795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.038692    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt\" (UniqueName: \"kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt\") pod \"31f0d812-72f4-488d-8487-72037df581eb\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") "
Nov 02 18:45:36.038795 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.038736    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca\") pod \"31f0d812-72f4-488d-8487-72037df581eb\" (UID: \"31f0d812-72f4-488d-8487-72037df581eb\") "
Nov 02 18:45:36.039220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.039186    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca" pod "31f0d812-72f4-488d-8487-72037df581eb" (UID: "31f0d812-72f4-488d-8487-72037df581eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:36.041063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.041027    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt" pod "31f0d812-72f4-488d-8487-72037df581eb" (UID: "31f0d812-72f4-488d-8487-72037df581eb"). InnerVolumeSpecName "kube-api-access-4ftjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:36.041206 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.041181    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca" pod "31f0d812-72f4-488d-8487-72037df581eb" (UID: "31f0d812-72f4-488d-8487-72037df581eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:45:36.139874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.139762    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftjt\" (UniqueName: \"kubernetes.io/projected/31f0d812-72f4-488d-8487-72037df581eb-kube-api-access-4ftjt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:36.139874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.139799    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f0d812-72f4-488d-8487-72037df581eb-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:36.139874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.139809    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f0d812-72f4-488d-8487-72037df581eb-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:36.852677 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.852642    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"
Nov 02 18:45:36.852891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.852674    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg" event={"ID":"31f0d812-72f4-488d-8487-72037df581eb","Type":"ContainerDied","Data":"908093d00c412e559243e0a2095d35d42b83591cecdcfb95f44e67cc821146ef"}
Nov 02 18:45:36.852891 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:36.852727    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908093d00c412e559243e0a2095d35d42b83591cecdcfb95f44e67cc821146ef"
Nov 02 18:45:48.331155 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:48.331059    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:45:48.331898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:48.331860    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:45:48.538557 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:45:48.538521    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:45:49.067017 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.066974    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"]
Nov 02 18:45:49.067434 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.067416    5042 cpu_manager.go:414] "containerMap: removing container" podUID="31f0d812-72f4-488d-8487-72037df581eb" containerName="revoke-keys"
Nov 02 18:45:49.067483 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.067457    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f0d812-72f4-488d-8487-72037df581eb" containerName="revoke-keys"
Nov 02 18:45:49.067524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.067481    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:49.067524 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.067488    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da1f5f5-19ce-46bc-972e-73732c8540a9" containerName="cleanup"
Nov 02 18:45:49.067597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.067589    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f0d812-72f4-488d-8487-72037df581eb" containerName="revoke-keys"
Nov 02 18:45:49.070882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.070862    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.076492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.076147    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:45:49.076492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.076342    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:45:49.077054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.077031    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:45:49.077828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.077607    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:45:49.078653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.078633    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"]
Nov 02 18:45:49.192631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.192589    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6jv\" (UniqueName: \"kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.192817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.192649    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.192817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.192780    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.294344 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.294307    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.294565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.294400    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.294565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.294497    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6jv\" (UniqueName: \"kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.295015 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.294988    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.297001 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.296980    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.302435 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.302408    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6jv\" (UniqueName: \"kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv\") pod \"maas-api-revoke-sub-2e1f5b5f-hqzzx\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.389650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.389562    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:49.667085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.667057    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"]
Nov 02 18:45:49.668992 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:45:49.668952    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2ffeff_aac2_4bfa_bae7_fa524cc9904d.slice/crio-26de44fb78377f5b5e2a7472ba48efadcbba0b1c1a80d8021f8e668c2a354a37 WatchSource:0}: Error finding container 26de44fb78377f5b5e2a7472ba48efadcbba0b1c1a80d8021f8e668c2a354a37: Status 404 returned error can't find the container with id 26de44fb78377f5b5e2a7472ba48efadcbba0b1c1a80d8021f8e668c2a354a37
Nov 02 18:45:49.960360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:49.960323    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx" event={"ID":"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d","Type":"ContainerStarted","Data":"26de44fb78377f5b5e2a7472ba48efadcbba0b1c1a80d8021f8e668c2a354a37"}
Nov 02 18:45:50.604491 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:45:50.604459    5042 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2ffeff_aac2_4bfa_bae7_fa524cc9904d.slice/crio-14a529fd9c823f130b1f4d2cf7988e62f41144179830dcc2fd57336b2fd1693b.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:45:50.967090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:50.967057    5042 generic.go:356] "Generic (PLEG): container finished" podID="cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" containerID="14a529fd9c823f130b1f4d2cf7988e62f41144179830dcc2fd57336b2fd1693b" exitCode=0
Nov 02 18:45:50.967302 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:50.967202    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx" event={"ID":"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d","Type":"ContainerDied","Data":"14a529fd9c823f130b1f4d2cf7988e62f41144179830dcc2fd57336b2fd1693b"}
Nov 02 18:45:52.126145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.126118    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:52.232031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.231941    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca\") pod \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") "
Nov 02 18:45:52.232031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.232028    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca\") pod \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") "
Nov 02 18:45:52.232270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.232105    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv\" (UniqueName: \"kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv\") pod \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\" (UID: \"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d\") "
Nov 02 18:45:52.232403 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.232374    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca" pod "cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" (UID: "cd2ffeff-aac2-4bfa-bae7-fa524cc9904d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:45:52.234393 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.234369    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca" pod "cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" (UID: "cd2ffeff-aac2-4bfa-bae7-fa524cc9904d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:45:52.234499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.234459    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv" pod "cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" (UID: "cd2ffeff-aac2-4bfa-bae7-fa524cc9904d"). InnerVolumeSpecName "kube-api-access-7v6jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:45:52.333443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.333403    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6jv\" (UniqueName: \"kubernetes.io/projected/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-kube-api-access-7v6jv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:52.333443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.333436    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:52.333443 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.333447    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:45:52.979455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.979427    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"
Nov 02 18:45:52.979628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.979465    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx" event={"ID":"cd2ffeff-aac2-4bfa-bae7-fa524cc9904d","Type":"ContainerDied","Data":"26de44fb78377f5b5e2a7472ba48efadcbba0b1c1a80d8021f8e668c2a354a37"}
Nov 02 18:45:52.979628 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:45:52.979522    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26de44fb78377f5b5e2a7472ba48efadcbba0b1c1a80d8021f8e668c2a354a37"
Nov 02 18:46:00.031822 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.031780    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"]
Nov 02 18:46:00.032337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.032281    5042 cpu_manager.go:414] "containerMap: removing container" podUID="cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" containerName="revoke-keys"
Nov 02 18:46:00.032337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.032293    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" containerName="revoke-keys"
Nov 02 18:46:00.032472 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.032393    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" containerName="revoke-keys"
Nov 02 18:46:00.036027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.036002    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.043985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.043950    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"]
Nov 02 18:46:00.044204 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.044146    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:46:00.044306 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.044244    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:46:00.044375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.044310    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:46:00.044375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.044367    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:46:00.114198 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.114162    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.114428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.114214    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.114428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.114341    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nn7\" (UniqueName: \"kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.215041 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.215001    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.215238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.215070    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.215238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.215144    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nn7\" (UniqueName: \"kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.215867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.215820    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.217654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.217628    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.223625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.223600    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nn7\" (UniqueName: \"kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7\") pod \"maas-api-revoke-sub-ec843018-szlvq\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.353215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.353108    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:00.642936 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:00.642912    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"]
Nov 02 18:46:00.645419 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:46:00.645372    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb6ba59_bffc_45d5_b346_56c458102f03.slice/crio-ec3790b39ef6814cd53492297785b5f14b1fcb9801da590697f243ce1a8d4668 WatchSource:0}: Error finding container ec3790b39ef6814cd53492297785b5f14b1fcb9801da590697f243ce1a8d4668: Status 404 returned error can't find the container with id ec3790b39ef6814cd53492297785b5f14b1fcb9801da590697f243ce1a8d4668
Nov 02 18:46:01.026054 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:01.026007    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq" event={"ID":"9cb6ba59-bffc-45d5-b346-56c458102f03","Type":"ContainerStarted","Data":"e1d6f04683bd9a06f88670a753d984d467ccdd2cf5e819892028d3cc6182606e"}
Nov 02 18:46:01.026264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:01.026079    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq" event={"ID":"9cb6ba59-bffc-45d5-b346-56c458102f03","Type":"ContainerStarted","Data":"ec3790b39ef6814cd53492297785b5f14b1fcb9801da590697f243ce1a8d4668"}
Nov 02 18:46:01.043992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:01.043921    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq" podStartSLOduration=1.043906305 podStartE2EDuration="1.043906305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:01.041592848 +0000 UTC m=+4373.621002236" watchObservedRunningTime="2026-11-02 18:46:01.043906305 +0000 UTC m=+4373.623315696"
Nov 02 18:46:02.038511 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:02.038473    5042 generic.go:356] "Generic (PLEG): container finished" podID="9cb6ba59-bffc-45d5-b346-56c458102f03" containerID="e1d6f04683bd9a06f88670a753d984d467ccdd2cf5e819892028d3cc6182606e" exitCode=0
Nov 02 18:46:02.038742 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:02.038566    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq" event={"ID":"9cb6ba59-bffc-45d5-b346-56c458102f03","Type":"ContainerDied","Data":"e1d6f04683bd9a06f88670a753d984d467ccdd2cf5e819892028d3cc6182606e"}
Nov 02 18:46:03.778713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.777283    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:03.895991 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.895890    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca\") pod \"9cb6ba59-bffc-45d5-b346-56c458102f03\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") "
Nov 02 18:46:03.896752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.896726    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7\" (UniqueName: \"kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7\") pod \"9cb6ba59-bffc-45d5-b346-56c458102f03\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") "
Nov 02 18:46:03.897295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.897268    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca\") pod \"9cb6ba59-bffc-45d5-b346-56c458102f03\" (UID: \"9cb6ba59-bffc-45d5-b346-56c458102f03\") "
Nov 02 18:46:03.898296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.896597    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca" pod "9cb6ba59-bffc-45d5-b346-56c458102f03" (UID: "9cb6ba59-bffc-45d5-b346-56c458102f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:03.903117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.903043    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7" pod "9cb6ba59-bffc-45d5-b346-56c458102f03" (UID: "9cb6ba59-bffc-45d5-b346-56c458102f03"). InnerVolumeSpecName "kube-api-access-v4nn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:03.905897 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.905829    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca" pod "9cb6ba59-bffc-45d5-b346-56c458102f03" (UID: "9cb6ba59-bffc-45d5-b346-56c458102f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:46:04.018867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.999604    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:04.018867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.999652    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cb6ba59-bffc-45d5-b346-56c458102f03-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:04.018867 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:03.999668    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4nn7\" (UniqueName: \"kubernetes.io/projected/9cb6ba59-bffc-45d5-b346-56c458102f03-kube-api-access-v4nn7\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:04.075863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:04.068095    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"
Nov 02 18:46:04.075863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:04.068141    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq" event={"ID":"9cb6ba59-bffc-45d5-b346-56c458102f03","Type":"ContainerDied","Data":"ec3790b39ef6814cd53492297785b5f14b1fcb9801da590697f243ce1a8d4668"}
Nov 02 18:46:04.075863 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:04.068209    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec3790b39ef6814cd53492297785b5f14b1fcb9801da590697f243ce1a8d4668"
Nov 02 18:46:18.539011 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:46:18.538975    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:46:23.152319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.152277    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"]
Nov 02 18:46:23.152764 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.152750    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9cb6ba59-bffc-45d5-b346-56c458102f03" containerName="revoke-keys"
Nov 02 18:46:23.152809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.152765    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb6ba59-bffc-45d5-b346-56c458102f03" containerName="revoke-keys"
Nov 02 18:46:23.152884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.152873    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb6ba59-bffc-45d5-b346-56c458102f03" containerName="revoke-keys"
Nov 02 18:46:23.156109 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.156094    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.162872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.160989    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:46:23.162872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.160990    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:46:23.162872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.160990    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:46:23.162872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.161411    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:46:23.163202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.162927    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"]
Nov 02 18:46:23.240803 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.240772    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scfpg\" (UniqueName: \"kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.241003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.240868    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.241003 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.240955    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.342310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.342269    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.342482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.342340    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scfpg\" (UniqueName: \"kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.342482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.342392    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.343183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.343109    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.344824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.344801    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.350232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.350212    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scfpg\" (UniqueName: \"kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg\") pod \"maas-api-revoke-sub-046e1343-zxqqs\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.475059 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.475024    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:23.749215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:23.749184    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"]
Nov 02 18:46:23.752206 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:46:23.752181    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c21f17_2083_4b0d_a1fa_30b67d26214f.slice/crio-2e57b42ee54f806b312d18faa7d6bed09a21a723af51b10cb34ab0a797d73fa1 WatchSource:0}: Error finding container 2e57b42ee54f806b312d18faa7d6bed09a21a723af51b10cb34ab0a797d73fa1: Status 404 returned error can't find the container with id 2e57b42ee54f806b312d18faa7d6bed09a21a723af51b10cb34ab0a797d73fa1
Nov 02 18:46:24.196336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:24.196290    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs" event={"ID":"36c21f17-2083-4b0d-a1fa-30b67d26214f","Type":"ContainerStarted","Data":"d2e4d93cdd198e546a96b0dead00790b47908ef391789224e13f5768edaa3157"}
Nov 02 18:46:24.196788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:24.196359    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs" event={"ID":"36c21f17-2083-4b0d-a1fa-30b67d26214f","Type":"ContainerStarted","Data":"2e57b42ee54f806b312d18faa7d6bed09a21a723af51b10cb34ab0a797d73fa1"}
Nov 02 18:46:24.214140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:24.214091    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs" podStartSLOduration=1.214076926 podStartE2EDuration="1.214076926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:24.212399012 +0000 UTC m=+4396.791808401" watchObservedRunningTime="2026-11-02 18:46:24.214076926 +0000 UTC m=+4396.793486303"
Nov 02 18:46:25.203093 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:25.203054    5042 generic.go:356] "Generic (PLEG): container finished" podID="36c21f17-2083-4b0d-a1fa-30b67d26214f" containerID="d2e4d93cdd198e546a96b0dead00790b47908ef391789224e13f5768edaa3157" exitCode=0
Nov 02 18:46:25.203492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:25.203142    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs" event={"ID":"36c21f17-2083-4b0d-a1fa-30b67d26214f","Type":"ContainerDied","Data":"d2e4d93cdd198e546a96b0dead00790b47908ef391789224e13f5768edaa3157"}
Nov 02 18:46:26.360228 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.360203    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:26.477579 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.477488    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca\") pod \"36c21f17-2083-4b0d-a1fa-30b67d26214f\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") "
Nov 02 18:46:26.477579 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.477576    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg\" (UniqueName: \"kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg\") pod \"36c21f17-2083-4b0d-a1fa-30b67d26214f\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") "
Nov 02 18:46:26.477780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.477631    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca\") pod \"36c21f17-2083-4b0d-a1fa-30b67d26214f\" (UID: \"36c21f17-2083-4b0d-a1fa-30b67d26214f\") "
Nov 02 18:46:26.478052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.478028    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca" pod "36c21f17-2083-4b0d-a1fa-30b67d26214f" (UID: "36c21f17-2083-4b0d-a1fa-30b67d26214f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:26.479938 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.479909    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca" pod "36c21f17-2083-4b0d-a1fa-30b67d26214f" (UID: "36c21f17-2083-4b0d-a1fa-30b67d26214f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:46:26.480056 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.480000    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg" pod "36c21f17-2083-4b0d-a1fa-30b67d26214f" (UID: "36c21f17-2083-4b0d-a1fa-30b67d26214f"). InnerVolumeSpecName "kube-api-access-scfpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:26.578563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.578526    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:26.578563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.578556    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scfpg\" (UniqueName: \"kubernetes.io/projected/36c21f17-2083-4b0d-a1fa-30b67d26214f-kube-api-access-scfpg\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:26.578563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:26.578567    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c21f17-2083-4b0d-a1fa-30b67d26214f-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:27.216807 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:27.216778    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"
Nov 02 18:46:27.217024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:27.216782    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs" event={"ID":"36c21f17-2083-4b0d-a1fa-30b67d26214f","Type":"ContainerDied","Data":"2e57b42ee54f806b312d18faa7d6bed09a21a723af51b10cb34ab0a797d73fa1"}
Nov 02 18:46:27.217024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:27.216913    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e57b42ee54f806b312d18faa7d6bed09a21a723af51b10cb34ab0a797d73fa1"
Nov 02 18:46:33.368038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.367997    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"]
Nov 02 18:46:33.368677 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.368659    5042 cpu_manager.go:414] "containerMap: removing container" podUID="36c21f17-2083-4b0d-a1fa-30b67d26214f" containerName="revoke-keys"
Nov 02 18:46:33.368724 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.368681    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c21f17-2083-4b0d-a1fa-30b67d26214f" containerName="revoke-keys"
Nov 02 18:46:33.368828 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.368816    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c21f17-2083-4b0d-a1fa-30b67d26214f" containerName="revoke-keys"
Nov 02 18:46:33.372882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.372859    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.377233 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.377205    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:46:33.378202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.378172    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:46:33.378348 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.378280    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:46:33.378413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.378353    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:46:33.379613 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.379583    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"]
Nov 02 18:46:33.441734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.441702    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.441734 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.441738    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.442024 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.441761    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmdv\" (UniqueName: \"kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.542737 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.542700    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.542737 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.542734    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.543051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.542753    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmdv\" (UniqueName: \"kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.543375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.543352    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.545301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.545273    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.550546 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.550521    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmdv\" (UniqueName: \"kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv\") pod \"maas-api-revoke-sub-d3fba056-nq558\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.694031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.693998    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:33.976476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:33.976441    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"]
Nov 02 18:46:34.192127 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:34.192097    5042 scope.go:122] "RemoveContainer" containerID="0576bb4c525a49639de375ab4a132580feb872f2e093c20ecf19b10b6c141871"
Nov 02 18:46:34.257351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:34.257264    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558" event={"ID":"70c223e0-4d82-4d7f-a872-cfa5568489c3","Type":"ContainerStarted","Data":"3ddb26cd220d024790c9b90b4fa46808f3a0650b1dda77a58010cb1133adb44e"}
Nov 02 18:46:35.301865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:35.299347    5042 generic.go:356] "Generic (PLEG): container finished" podID="70c223e0-4d82-4d7f-a872-cfa5568489c3" containerID="77d22f0c963fd295695b1a392cf5ba7c02956a191b266ec726956115f7f56e9b" exitCode=0
Nov 02 18:46:35.301865 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:35.299468    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558" event={"ID":"70c223e0-4d82-4d7f-a872-cfa5568489c3","Type":"ContainerDied","Data":"77d22f0c963fd295695b1a392cf5ba7c02956a191b266ec726956115f7f56e9b"}
Nov 02 18:46:36.858447 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.858414    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:36.963199 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.963160    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca\") pod \"70c223e0-4d82-4d7f-a872-cfa5568489c3\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") "
Nov 02 18:46:36.963406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.963313    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca\") pod \"70c223e0-4d82-4d7f-a872-cfa5568489c3\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") "
Nov 02 18:46:36.963406 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.963345    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv\" (UniqueName: \"kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv\") pod \"70c223e0-4d82-4d7f-a872-cfa5568489c3\" (UID: \"70c223e0-4d82-4d7f-a872-cfa5568489c3\") "
Nov 02 18:46:36.963752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.963717    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca" pod "70c223e0-4d82-4d7f-a872-cfa5568489c3" (UID: "70c223e0-4d82-4d7f-a872-cfa5568489c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:36.965651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.965621    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca" pod "70c223e0-4d82-4d7f-a872-cfa5568489c3" (UID: "70c223e0-4d82-4d7f-a872-cfa5568489c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:46:36.965793 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:36.965766    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv" pod "70c223e0-4d82-4d7f-a872-cfa5568489c3" (UID: "70c223e0-4d82-4d7f-a872-cfa5568489c3"). InnerVolumeSpecName "kube-api-access-5tmdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:37.064310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:37.064226    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:37.064310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:37.064257    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70c223e0-4d82-4d7f-a872-cfa5568489c3-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:37.064310 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:37.064268    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmdv\" (UniqueName: \"kubernetes.io/projected/70c223e0-4d82-4d7f-a872-cfa5568489c3-kube-api-access-5tmdv\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:37.349750 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:37.349671    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"
Nov 02 18:46:37.349944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:37.349671    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558" event={"ID":"70c223e0-4d82-4d7f-a872-cfa5568489c3","Type":"ContainerDied","Data":"3ddb26cd220d024790c9b90b4fa46808f3a0650b1dda77a58010cb1133adb44e"}
Nov 02 18:46:37.349944 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:37.349808    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ddb26cd220d024790c9b90b4fa46808f3a0650b1dda77a58010cb1133adb44e"
Nov 02 18:46:48.618869 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:46:48.618569    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:46:54.096951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.096915    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"]
Nov 02 18:46:54.097387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.097373    5042 cpu_manager.go:414] "containerMap: removing container" podUID="70c223e0-4d82-4d7f-a872-cfa5568489c3" containerName="revoke-keys"
Nov 02 18:46:54.097431 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.097387    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c223e0-4d82-4d7f-a872-cfa5568489c3" containerName="revoke-keys"
Nov 02 18:46:54.097486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.097476    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c223e0-4d82-4d7f-a872-cfa5568489c3" containerName="revoke-keys"
Nov 02 18:46:54.100862 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.100828    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.105201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.105121    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:46:54.105705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.105663    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:46:54.108824 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.108779    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"]
Nov 02 18:46:54.110165 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.110141    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:46:54.110324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.110254    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:46:54.184274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.184229    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.184488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.184385    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.184488 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.184446    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl66t\" (UniqueName: \"kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.285287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.285245    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.285486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.285355    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.285486 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.285384    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl66t\" (UniqueName: \"kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.286168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.286146    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.288014 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.287993    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.293541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.293514    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl66t\" (UniqueName: \"kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t\") pod \"maas-api-revoke-sub-ae594979-4qtxn\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.421930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.421802    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:54.706098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:54.706023    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"]
Nov 02 18:46:54.708168 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:46:54.708136    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1de37c1_3a4f_4aff_a71a_15188f2f8816.slice/crio-462e9b460a9df8771330d8f23c16fb14f8f5db532011b003dce12654a86c67fd WatchSource:0}: Error finding container 462e9b460a9df8771330d8f23c16fb14f8f5db532011b003dce12654a86c67fd: Status 404 returned error can't find the container with id 462e9b460a9df8771330d8f23c16fb14f8f5db532011b003dce12654a86c67fd
Nov 02 18:46:55.514663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:55.514626    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn" event={"ID":"a1de37c1-3a4f-4aff-a71a-15188f2f8816","Type":"ContainerStarted","Data":"774de9c8bd23582e292a50ba0222fd8bc34e4cafd155717f1e650a6ae0fcb993"}
Nov 02 18:46:55.515097 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:55.514688    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn" event={"ID":"a1de37c1-3a4f-4aff-a71a-15188f2f8816","Type":"ContainerStarted","Data":"462e9b460a9df8771330d8f23c16fb14f8f5db532011b003dce12654a86c67fd"}
Nov 02 18:46:55.530918 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:55.530830    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn" podStartSLOduration=1.530815021 podStartE2EDuration="1.530815021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:55.529887148 +0000 UTC m=+4428.109296526" watchObservedRunningTime="2026-11-02 18:46:55.530815021 +0000 UTC m=+4428.110224407"
Nov 02 18:46:56.521279 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:56.521246    5042 generic.go:356] "Generic (PLEG): container finished" podID="a1de37c1-3a4f-4aff-a71a-15188f2f8816" containerID="774de9c8bd23582e292a50ba0222fd8bc34e4cafd155717f1e650a6ae0fcb993" exitCode=0
Nov 02 18:46:56.521720 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:56.521339    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn" event={"ID":"a1de37c1-3a4f-4aff-a71a-15188f2f8816","Type":"ContainerDied","Data":"774de9c8bd23582e292a50ba0222fd8bc34e4cafd155717f1e650a6ae0fcb993"}
Nov 02 18:46:57.682782 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.682755    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:57.823124 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.823029    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca\") pod \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") "
Nov 02 18:46:57.823328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.823176    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t\" (UniqueName: \"kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t\") pod \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") "
Nov 02 18:46:57.823328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.823210    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca\") pod \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\" (UID: \"a1de37c1-3a4f-4aff-a71a-15188f2f8816\") "
Nov 02 18:46:57.823477 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.823448    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca" pod "a1de37c1-3a4f-4aff-a71a-15188f2f8816" (UID: "a1de37c1-3a4f-4aff-a71a-15188f2f8816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:46:57.825672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.825643    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca" pod "a1de37c1-3a4f-4aff-a71a-15188f2f8816" (UID: "a1de37c1-3a4f-4aff-a71a-15188f2f8816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:46:57.825672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.825656    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t" pod "a1de37c1-3a4f-4aff-a71a-15188f2f8816" (UID: "a1de37c1-3a4f-4aff-a71a-15188f2f8816"). InnerVolumeSpecName "kube-api-access-pl66t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:46:57.924507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.924457    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl66t\" (UniqueName: \"kubernetes.io/projected/a1de37c1-3a4f-4aff-a71a-15188f2f8816-kube-api-access-pl66t\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:57.924507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.924500    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:57.924507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:57.924510    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1de37c1-3a4f-4aff-a71a-15188f2f8816-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:46:58.534042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:58.534007    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"
Nov 02 18:46:58.534251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:58.534041    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn" event={"ID":"a1de37c1-3a4f-4aff-a71a-15188f2f8816","Type":"ContainerDied","Data":"462e9b460a9df8771330d8f23c16fb14f8f5db532011b003dce12654a86c67fd"}
Nov 02 18:46:58.534251 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:46:58.534089    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="462e9b460a9df8771330d8f23c16fb14f8f5db532011b003dce12654a86c67fd"
Nov 02 18:47:09.056894 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:09.047998    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"]
Nov 02 18:47:09.056894 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:09.053502    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-hp6mt"]
Nov 02 18:47:09.550695 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:09.550657    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b477d1-e4b3-4f5b-99f7-d7badc83c63d" path="/var/lib/kubelet/pods/c3b477d1-e4b3-4f5b-99f7-d7badc83c63d/volumes"
Nov 02 18:47:10.206989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.206945    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"]
Nov 02 18:47:10.207516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.207456    5042 cpu_manager.go:414] "containerMap: removing container" podUID="a1de37c1-3a4f-4aff-a71a-15188f2f8816" containerName="revoke-keys"
Nov 02 18:47:10.207516 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.207469    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1de37c1-3a4f-4aff-a71a-15188f2f8816" containerName="revoke-keys"
Nov 02 18:47:10.207646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.207570    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1de37c1-3a4f-4aff-a71a-15188f2f8816" containerName="revoke-keys"
Nov 02 18:47:10.212208 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.212181    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.216542 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.216515    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:47:10.217996 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.217916    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:47:10.218201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.218075    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:47:10.218201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.218121    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"]
Nov 02 18:47:10.218395 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.218162    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:47:10.276964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.276926    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncct\" (UniqueName: \"kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.277170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.276991    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.277170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.277105    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.378094 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.378051    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.378296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.378197    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncct\" (UniqueName: \"kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.378296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.378243    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.379055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.378974    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.381027 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.380995    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.387203 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.387169    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncct\" (UniqueName: \"kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct\") pod \"maas-api-revoke-sub-a4aef7d7-67jxz\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.539649 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.539555    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:10.825608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:10.825576    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"]
Nov 02 18:47:10.829361 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:47:10.829301    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cae6cf_f27e_4198_b65d_aa2e190e6d01.slice/crio-af66db1c77de3393b976aa234a102b90c8f6f19f5913b143547171c874452dc4 WatchSource:0}: Error finding container af66db1c77de3393b976aa234a102b90c8f6f19f5913b143547171c874452dc4: Status 404 returned error can't find the container with id af66db1c77de3393b976aa234a102b90c8f6f19f5913b143547171c874452dc4
Nov 02 18:47:11.644292 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:11.644246    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz" event={"ID":"78cae6cf-f27e-4198-b65d-aa2e190e6d01","Type":"ContainerStarted","Data":"38a5aa2512a34e9d30eee4de6010b3f5134666a28ae42d64c38ff2bed85d18c9"}
Nov 02 18:47:11.644728 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:11.644317    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz" event={"ID":"78cae6cf-f27e-4198-b65d-aa2e190e6d01","Type":"ContainerStarted","Data":"af66db1c77de3393b976aa234a102b90c8f6f19f5913b143547171c874452dc4"}
Nov 02 18:47:11.662086 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:11.662027    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz" podStartSLOduration=1.6620092770000001 podStartE2EDuration="1.662009277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:11.659738086 +0000 UTC m=+4444.239147473" watchObservedRunningTime="2026-11-02 18:47:11.662009277 +0000 UTC m=+4444.241418663"
Nov 02 18:47:12.652293 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:12.652249    5042 generic.go:356] "Generic (PLEG): container finished" podID="78cae6cf-f27e-4198-b65d-aa2e190e6d01" containerID="38a5aa2512a34e9d30eee4de6010b3f5134666a28ae42d64c38ff2bed85d18c9" exitCode=0
Nov 02 18:47:12.652754 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:12.652326    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz" event={"ID":"78cae6cf-f27e-4198-b65d-aa2e190e6d01","Type":"ContainerDied","Data":"38a5aa2512a34e9d30eee4de6010b3f5134666a28ae42d64c38ff2bed85d18c9"}
Nov 02 18:47:13.827886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.827817    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:13.914939 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.914819    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct\" (UniqueName: \"kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct\") pod \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") "
Nov 02 18:47:13.915166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.914968    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca\") pod \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") "
Nov 02 18:47:13.915166 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.915037    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca\") pod \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\" (UID: \"78cae6cf-f27e-4198-b65d-aa2e190e6d01\") "
Nov 02 18:47:13.915499 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.915466    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca" pod "78cae6cf-f27e-4198-b65d-aa2e190e6d01" (UID: "78cae6cf-f27e-4198-b65d-aa2e190e6d01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:47:13.917355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.917327    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca" pod "78cae6cf-f27e-4198-b65d-aa2e190e6d01" (UID: "78cae6cf-f27e-4198-b65d-aa2e190e6d01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:13.917479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:13.917328    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct" pod "78cae6cf-f27e-4198-b65d-aa2e190e6d01" (UID: "78cae6cf-f27e-4198-b65d-aa2e190e6d01"). InnerVolumeSpecName "kube-api-access-hncct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:14.016717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:14.016667    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:14.016717 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:14.016715    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78cae6cf-f27e-4198-b65d-aa2e190e6d01-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:14.016994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:14.016731    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hncct\" (UniqueName: \"kubernetes.io/projected/78cae6cf-f27e-4198-b65d-aa2e190e6d01-kube-api-access-hncct\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:14.667040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:14.667000    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz" event={"ID":"78cae6cf-f27e-4198-b65d-aa2e190e6d01","Type":"ContainerDied","Data":"af66db1c77de3393b976aa234a102b90c8f6f19f5913b143547171c874452dc4"}
Nov 02 18:47:14.667040 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:14.667040    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"
Nov 02 18:47:14.667334 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:14.667065    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af66db1c77de3393b976aa234a102b90c8f6f19f5913b143547171c874452dc4"
Nov 02 18:47:18.539982 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:47:18.539895    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:47:23.025066 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:23.025014    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"]
Nov 02 18:47:23.028057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:23.028021    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-2d9x7"]
Nov 02 18:47:23.546529 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:23.546479    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef4a0aba-7029-4d01-afef-a78c0ea17c56" path="/var/lib/kubelet/pods/ef4a0aba-7029-4d01-afef-a78c0ea17c56/volumes"
Nov 02 18:47:25.662120 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.662070    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"]
Nov 02 18:47:25.662818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.662794    5042 cpu_manager.go:414] "containerMap: removing container" podUID="78cae6cf-f27e-4198-b65d-aa2e190e6d01" containerName="revoke-keys"
Nov 02 18:47:25.662951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.662820    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cae6cf-f27e-4198-b65d-aa2e190e6d01" containerName="revoke-keys"
Nov 02 18:47:25.663016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.662994    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cae6cf-f27e-4198-b65d-aa2e190e6d01" containerName="revoke-keys"
Nov 02 18:47:25.667035 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.667014    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.671132 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.671100    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"]
Nov 02 18:47:25.671884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.671782    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:47:25.671884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.671798    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:47:25.671884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.671805    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:47:25.672140 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.671923    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:47:25.763916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.763504    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5zf\" (UniqueName: \"kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.763916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.763607    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.763916 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.763696    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.868866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.865904    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.868866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.866013    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5zf\" (UniqueName: \"kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.868866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.866096    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.868866 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.867964    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.869725 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.869693    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:25.918745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:25.918616    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5zf\" (UniqueName: \"kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf\") pod \"maas-api-revoke-sub-79805ca9-f724k\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:26.066666 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:26.066615    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:26.390783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:26.390750    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"]
Nov 02 18:47:26.394368 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:47:26.394338    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dd54e2_0b88_443f_bdf4_43c4b14a89da.slice/crio-af0603d6a216d8ea18805098ab8d7364e9f383bb430be84e1f089163a6147032 WatchSource:0}: Error finding container af0603d6a216d8ea18805098ab8d7364e9f383bb430be84e1f089163a6147032: Status 404 returned error can't find the container with id af0603d6a216d8ea18805098ab8d7364e9f383bb430be84e1f089163a6147032
Nov 02 18:47:26.771759 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:26.771718    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k" event={"ID":"31dd54e2-0b88-443f-bdf4-43c4b14a89da","Type":"ContainerStarted","Data":"4b54432a4639f3c24564bc043f6e7a50b29e5f05caefb4867f4894d409474a9b"}
Nov 02 18:47:26.772274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:26.771780    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k" event={"ID":"31dd54e2-0b88-443f-bdf4-43c4b14a89da","Type":"ContainerStarted","Data":"af0603d6a216d8ea18805098ab8d7364e9f383bb430be84e1f089163a6147032"}
Nov 02 18:47:26.790267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:26.790202    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k" podStartSLOduration=1.790186946 podStartE2EDuration="1.790186946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:26.788336266 +0000 UTC m=+4459.367745652" watchObservedRunningTime="2026-11-02 18:47:26.790186946 +0000 UTC m=+4459.369596331"
Nov 02 18:47:27.779606 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:27.779572    5042 generic.go:356] "Generic (PLEG): container finished" podID="31dd54e2-0b88-443f-bdf4-43c4b14a89da" containerID="4b54432a4639f3c24564bc043f6e7a50b29e5f05caefb4867f4894d409474a9b" exitCode=0
Nov 02 18:47:27.780132 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:27.779610    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k" event={"ID":"31dd54e2-0b88-443f-bdf4-43c4b14a89da","Type":"ContainerDied","Data":"4b54432a4639f3c24564bc043f6e7a50b29e5f05caefb4867f4894d409474a9b"}
Nov 02 18:47:28.944378 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:28.944346    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:29.014294 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.014254    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf\" (UniqueName: \"kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf\") pod \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") "
Nov 02 18:47:29.014465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.014316    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca\") pod \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") "
Nov 02 18:47:29.014465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.014418    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca\") pod \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\" (UID: \"31dd54e2-0b88-443f-bdf4-43c4b14a89da\") "
Nov 02 18:47:29.014660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.014640    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca" pod "31dd54e2-0b88-443f-bdf4-43c4b14a89da" (UID: "31dd54e2-0b88-443f-bdf4-43c4b14a89da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:47:29.014818 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.014802    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:29.016743 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.016719    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca" pod "31dd54e2-0b88-443f-bdf4-43c4b14a89da" (UID: "31dd54e2-0b88-443f-bdf4-43c4b14a89da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:29.016926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.016908    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf" pod "31dd54e2-0b88-443f-bdf4-43c4b14a89da" (UID: "31dd54e2-0b88-443f-bdf4-43c4b14a89da"). InnerVolumeSpecName "kube-api-access-sn5zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:29.115620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.115526    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5zf\" (UniqueName: \"kubernetes.io/projected/31dd54e2-0b88-443f-bdf4-43c4b14a89da-kube-api-access-sn5zf\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:29.115620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.115562    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31dd54e2-0b88-443f-bdf4-43c4b14a89da-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:29.793070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.793031    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"
Nov 02 18:47:29.793267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.793065    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k" event={"ID":"31dd54e2-0b88-443f-bdf4-43c4b14a89da","Type":"ContainerDied","Data":"af0603d6a216d8ea18805098ab8d7364e9f383bb430be84e1f089163a6147032"}
Nov 02 18:47:29.793267 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:29.793120    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0603d6a216d8ea18805098ab8d7364e9f383bb430be84e1f089163a6147032"
Nov 02 18:47:34.221700 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:34.221667    5042 scope.go:122] "RemoveContainer" containerID="8d0c992604d255eb981fe5b8ab311bae6bb33fc6acbf6af334c4dbd86f0e2985"
Nov 02 18:47:39.024373 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:39.024332    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"]
Nov 02 18:47:39.029879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:39.029801    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c08e66-24ml9"]
Nov 02 18:47:39.553706 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:39.553654    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0a3daf-4171-45b6-95de-c97bfeb6ce08" path="/var/lib/kubelet/pods/7e0a3daf-4171-45b6-95de-c97bfeb6ce08/volumes"
Nov 02 18:47:40.297941 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.297895    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"]
Nov 02 18:47:40.307433 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.307392    5042 cpu_manager.go:414] "containerMap: removing container" podUID="31dd54e2-0b88-443f-bdf4-43c4b14a89da" containerName="revoke-keys"
Nov 02 18:47:40.307678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.307660    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dd54e2-0b88-443f-bdf4-43c4b14a89da" containerName="revoke-keys"
Nov 02 18:47:40.307989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.307971    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dd54e2-0b88-443f-bdf4-43c4b14a89da" containerName="revoke-keys"
Nov 02 18:47:40.313043 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.313015    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"]
Nov 02 18:47:40.313337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.313321    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.321568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.321531    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:47:40.324322 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.324181    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:47:40.327337 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.327044    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:47:40.327541 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.327176    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:47:40.453047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.452664    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxp2\" (UniqueName: \"kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.453047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.452762    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.453047 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.452898    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.554012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.553896    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxp2\" (UniqueName: \"kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.554012 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.554003    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.554311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.554099    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.554988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.554956    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.557919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.557889    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.564296 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.564258    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxp2\" (UniqueName: \"kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2\") pod \"maas-api-revoke-sub-0b7e8edb-4cmpq\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:40.667160 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:40.667122    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:41.199394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:41.199345    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"]
Nov 02 18:47:41.927360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:41.927256    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq" event={"ID":"458ad5d6-9ea1-471b-bbf3-e13056b7dc58","Type":"ContainerStarted","Data":"ee5c0ddbfbd287283260896574868b551be8e82dc7b5595f633950eda081d3e4"}
Nov 02 18:47:41.927360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:41.927327    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq" event={"ID":"458ad5d6-9ea1-471b-bbf3-e13056b7dc58","Type":"ContainerStarted","Data":"b7bab94498182c99eec4d6a66aef8e0154cd408edc6c0047f4959b6981db5de6"}
Nov 02 18:47:41.944081 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:41.944025    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq" podStartSLOduration=1.944011688 podStartE2EDuration="1.944011688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:41.942702437 +0000 UTC m=+4474.522111822" watchObservedRunningTime="2026-11-02 18:47:41.944011688 +0000 UTC m=+4474.523421078"
Nov 02 18:47:42.937347 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:42.937312    5042 generic.go:356] "Generic (PLEG): container finished" podID="458ad5d6-9ea1-471b-bbf3-e13056b7dc58" containerID="ee5c0ddbfbd287283260896574868b551be8e82dc7b5595f633950eda081d3e4" exitCode=0
Nov 02 18:47:42.937833 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:42.937398    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq" event={"ID":"458ad5d6-9ea1-471b-bbf3-e13056b7dc58","Type":"ContainerDied","Data":"ee5c0ddbfbd287283260896574868b551be8e82dc7b5595f633950eda081d3e4"}
Nov 02 18:47:44.092568 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.092539    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:44.195984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.195881    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca\") pod \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") "
Nov 02 18:47:44.195984 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.195959    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2\" (UniqueName: \"kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2\") pod \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") "
Nov 02 18:47:44.196247 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.196051    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca\") pod \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\" (UID: \"458ad5d6-9ea1-471b-bbf3-e13056b7dc58\") "
Nov 02 18:47:44.196324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.196301    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca" pod "458ad5d6-9ea1-471b-bbf3-e13056b7dc58" (UID: "458ad5d6-9ea1-471b-bbf3-e13056b7dc58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:47:44.198414 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.198384    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2" pod "458ad5d6-9ea1-471b-bbf3-e13056b7dc58" (UID: "458ad5d6-9ea1-471b-bbf3-e13056b7dc58"). InnerVolumeSpecName "kube-api-access-kqxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:44.198533 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.198431    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca" pod "458ad5d6-9ea1-471b-bbf3-e13056b7dc58" (UID: "458ad5d6-9ea1-471b-bbf3-e13056b7dc58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:44.297076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.297031    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:44.297076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.297072    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:44.297309 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.297089    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxp2\" (UniqueName: \"kubernetes.io/projected/458ad5d6-9ea1-471b-bbf3-e13056b7dc58-kube-api-access-kqxp2\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:44.950632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.950593    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"
Nov 02 18:47:44.950632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.950616    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq" event={"ID":"458ad5d6-9ea1-471b-bbf3-e13056b7dc58","Type":"ContainerDied","Data":"b7bab94498182c99eec4d6a66aef8e0154cd408edc6c0047f4959b6981db5de6"}
Nov 02 18:47:44.950921 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:44.950681    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7bab94498182c99eec4d6a66aef8e0154cd408edc6c0047f4959b6981db5de6"
Nov 02 18:47:48.540340 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:47:48.540294    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:47:50.937276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.937234    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"]
Nov 02 18:47:50.937973 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.937949    5042 cpu_manager.go:414] "containerMap: removing container" podUID="458ad5d6-9ea1-471b-bbf3-e13056b7dc58" containerName="revoke-keys"
Nov 02 18:47:50.938075 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.937975    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458ad5d6-9ea1-471b-bbf3-e13056b7dc58" containerName="revoke-keys"
Nov 02 18:47:50.938153 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.938141    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="458ad5d6-9ea1-471b-bbf3-e13056b7dc58" containerName="revoke-keys"
Nov 02 18:47:50.941453 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.941418    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:50.946581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.946553    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:47:50.946757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.946617    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:47:50.948038 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.948016    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:47:50.948351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.948320    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:47:50.948872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.948817    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"]
Nov 02 18:47:50.965709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.965659    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2g75\" (UniqueName: \"kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:50.965913 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.965812    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:50.965988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:50.965924    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.066912 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.066864    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2g75\" (UniqueName: \"kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.067117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.066934    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.067117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.066968    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.067678 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.067654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.069688 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.069654    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.075411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.075382    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2g75\" (UniqueName: \"kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75\") pod \"maas-api-revoke-sub-3d4156f1-fxwjr\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.258901 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.258827    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:51.538967 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:47:51.538934    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aa4772_fef3_4828_a32f_01ab269ee77f.slice/crio-49b127588b942fc578ea17eea57ea8239a515b6e82326bb6fb574057ee52f102 WatchSource:0}: Error finding container 49b127588b942fc578ea17eea57ea8239a515b6e82326bb6fb574057ee52f102: Status 404 returned error can't find the container with id 49b127588b942fc578ea17eea57ea8239a515b6e82326bb6fb574057ee52f102
Nov 02 18:47:51.543694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.543669    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"]
Nov 02 18:47:51.994607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.994568    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr" event={"ID":"90aa4772-fef3-4828-a32f-01ab269ee77f","Type":"ContainerStarted","Data":"6c0ac25d106b3204bef6c4a61d8fe157532934d46cd3f2d1aa426c7fa7ebfe60"}
Nov 02 18:47:51.995052 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:51.994620    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr" event={"ID":"90aa4772-fef3-4828-a32f-01ab269ee77f","Type":"ContainerStarted","Data":"49b127588b942fc578ea17eea57ea8239a515b6e82326bb6fb574057ee52f102"}
Nov 02 18:47:52.011260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:52.011203    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr" podStartSLOduration=2.0111874260000002 podStartE2EDuration="2.011187426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:52.009647184 +0000 UTC m=+4484.589056571" watchObservedRunningTime="2026-11-02 18:47:52.011187426 +0000 UTC m=+4484.590596811"
Nov 02 18:47:53.009432 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:53.009397    5042 generic.go:356] "Generic (PLEG): container finished" podID="90aa4772-fef3-4828-a32f-01ab269ee77f" containerID="6c0ac25d106b3204bef6c4a61d8fe157532934d46cd3f2d1aa426c7fa7ebfe60" exitCode=0
Nov 02 18:47:53.009929 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:53.009492    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr" event={"ID":"90aa4772-fef3-4828-a32f-01ab269ee77f","Type":"ContainerDied","Data":"6c0ac25d106b3204bef6c4a61d8fe157532934d46cd3f2d1aa426c7fa7ebfe60"}
Nov 02 18:47:54.168362 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.168334    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:54.200534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.200501    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca\") pod \"90aa4772-fef3-4828-a32f-01ab269ee77f\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") "
Nov 02 18:47:54.200736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.200565    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca\") pod \"90aa4772-fef3-4828-a32f-01ab269ee77f\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") "
Nov 02 18:47:54.200736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.200609    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75\" (UniqueName: \"kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75\") pod \"90aa4772-fef3-4828-a32f-01ab269ee77f\" (UID: \"90aa4772-fef3-4828-a32f-01ab269ee77f\") "
Nov 02 18:47:54.200898 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.200874    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca" pod "90aa4772-fef3-4828-a32f-01ab269ee77f" (UID: "90aa4772-fef3-4828-a32f-01ab269ee77f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:47:54.201090 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.201072    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:54.203694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.203654    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca" pod "90aa4772-fef3-4828-a32f-01ab269ee77f" (UID: "90aa4772-fef3-4828-a32f-01ab269ee77f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:47:54.204088 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.204064    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75" pod "90aa4772-fef3-4828-a32f-01ab269ee77f" (UID: "90aa4772-fef3-4828-a32f-01ab269ee77f"). InnerVolumeSpecName "kube-api-access-j2g75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:47:54.302625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.302531    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90aa4772-fef3-4828-a32f-01ab269ee77f-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:54.302625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:54.302568    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2g75\" (UniqueName: \"kubernetes.io/projected/90aa4772-fef3-4828-a32f-01ab269ee77f-kube-api-access-j2g75\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:47:55.022789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:55.022756    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"
Nov 02 18:47:55.023031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:55.022795    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr" event={"ID":"90aa4772-fef3-4828-a32f-01ab269ee77f","Type":"ContainerDied","Data":"49b127588b942fc578ea17eea57ea8239a515b6e82326bb6fb574057ee52f102"}
Nov 02 18:47:55.023031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:47:55.022866    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49b127588b942fc578ea17eea57ea8239a515b6e82326bb6fb574057ee52f102"
Nov 02 18:48:07.572237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.572199    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"]
Nov 02 18:48:07.576260 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.576221    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fxwjr"]
Nov 02 18:48:07.605148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.605116    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"]
Nov 02 18:48:07.605647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.605619    5042 cpu_manager.go:414] "containerMap: removing container" podUID="90aa4772-fef3-4828-a32f-01ab269ee77f" containerName="revoke-keys"
Nov 02 18:48:07.605647 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.605640    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90aa4772-fef3-4828-a32f-01ab269ee77f" containerName="revoke-keys"
Nov 02 18:48:07.605904 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.605774    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="90aa4772-fef3-4828-a32f-01ab269ee77f" containerName="revoke-keys"
Nov 02 18:48:07.609384 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.609361    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.614117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.614091    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:48:07.614382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.614364    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:48:07.615084 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.615060    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:48:07.615731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.615714    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:48:07.616545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.616521    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"]
Nov 02 18:48:07.695976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.695938    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.696168 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.696084    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.696215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.696161    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7267r\" (UniqueName: \"kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.797739 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.797698    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.797978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.797769    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.797978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.797795    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7267r\" (UniqueName: \"kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.798470 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.798447    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.800557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.800535    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:48:07.807591 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.807559    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7267r\" (UniqueName: \"kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.810412 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.810380    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nk9tk\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:07.930237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.930153    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:48:07.938272 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:07.938221    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:08.223273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:08.223239    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"]
Nov 02 18:48:08.225222 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:48:08.225192    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636a850c_6e08_4558_a9ee_409e993dc075.slice/crio-32de98e67ae3027deca4b50c80f51a2ea393ba04d125983364568f2fd4add539 WatchSource:0}: Error finding container 32de98e67ae3027deca4b50c80f51a2ea393ba04d125983364568f2fd4add539: Status 404 returned error can't find the container with id 32de98e67ae3027deca4b50c80f51a2ea393ba04d125983364568f2fd4add539
Nov 02 18:48:09.163922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:09.163880    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk" event={"ID":"636a850c-6e08-4558-a9ee-409e993dc075","Type":"ContainerStarted","Data":"0edc054771db1a35a4a8e995d5bac5fb93c4509194ccd8300bbbf9c2bf279ef3"}
Nov 02 18:48:09.164415 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:09.163942    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk" event={"ID":"636a850c-6e08-4558-a9ee-409e993dc075","Type":"ContainerStarted","Data":"32de98e67ae3027deca4b50c80f51a2ea393ba04d125983364568f2fd4add539"}
Nov 02 18:48:09.180610 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:09.180562    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk" podStartSLOduration=2.180548575 podStartE2EDuration="2.180548575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:48:09.180050644 +0000 UTC m=+4501.759460029" watchObservedRunningTime="2026-11-02 18:48:09.180548575 +0000 UTC m=+4501.759957960"
Nov 02 18:48:09.551518 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:09.551479    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90aa4772-fef3-4828-a32f-01ab269ee77f" path="/var/lib/kubelet/pods/90aa4772-fef3-4828-a32f-01ab269ee77f/volumes"
Nov 02 18:48:10.003561 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:10.003536    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:48:10.003804 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:10.003596    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:48:10.006532 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:10.006509    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:48:10.006683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:10.006586    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:48:10.172079 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:10.172044    5042 generic.go:356] "Generic (PLEG): container finished" podID="636a850c-6e08-4558-a9ee-409e993dc075" containerID="0edc054771db1a35a4a8e995d5bac5fb93c4509194ccd8300bbbf9c2bf279ef3" exitCode=0
Nov 02 18:48:10.172498 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:10.172113    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk" event={"ID":"636a850c-6e08-4558-a9ee-409e993dc075","Type":"ContainerDied","Data":"0edc054771db1a35a4a8e995d5bac5fb93c4509194ccd8300bbbf9c2bf279ef3"}
Nov 02 18:48:11.326626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.326596    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:11.438788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.438753    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca\") pod \"636a850c-6e08-4558-a9ee-409e993dc075\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") "
Nov 02 18:48:11.439046 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.438878    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r\" (UniqueName: \"kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r\") pod \"636a850c-6e08-4558-a9ee-409e993dc075\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") "
Nov 02 18:48:11.439046 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.438914    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca\") pod \"636a850c-6e08-4558-a9ee-409e993dc075\" (UID: \"636a850c-6e08-4558-a9ee-409e993dc075\") "
Nov 02 18:48:11.439170 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.439146    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca" pod "636a850c-6e08-4558-a9ee-409e993dc075" (UID: "636a850c-6e08-4558-a9ee-409e993dc075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:11.439342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.439328    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:11.441319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.441288    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca" pod "636a850c-6e08-4558-a9ee-409e993dc075" (UID: "636a850c-6e08-4558-a9ee-409e993dc075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:11.441436 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.441324    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r" pod "636a850c-6e08-4558-a9ee-409e993dc075" (UID: "636a850c-6e08-4558-a9ee-409e993dc075"). InnerVolumeSpecName "kube-api-access-7267r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:11.539976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.539943    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7267r\" (UniqueName: \"kubernetes.io/projected/636a850c-6e08-4558-a9ee-409e993dc075-kube-api-access-7267r\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:11.539976 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:11.539969    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a850c-6e08-4558-a9ee-409e993dc075-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:12.185253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:12.185225    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"
Nov 02 18:48:12.185469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:12.185269    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk" event={"ID":"636a850c-6e08-4558-a9ee-409e993dc075","Type":"ContainerDied","Data":"32de98e67ae3027deca4b50c80f51a2ea393ba04d125983364568f2fd4add539"}
Nov 02 18:48:12.185469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:12.185325    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32de98e67ae3027deca4b50c80f51a2ea393ba04d125983364568f2fd4add539"
Nov 02 18:48:18.540538 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:48:18.540499    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:48:29.978700 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.978655    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"]
Nov 02 18:48:29.979215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.979172    5042 cpu_manager.go:414] "containerMap: removing container" podUID="636a850c-6e08-4558-a9ee-409e993dc075" containerName="revoke-keys"
Nov 02 18:48:29.979215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.979186    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636a850c-6e08-4558-a9ee-409e993dc075" containerName="revoke-keys"
Nov 02 18:48:29.979301 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.979259    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="636a850c-6e08-4558-a9ee-409e993dc075" containerName="revoke-keys"
Nov 02 18:48:29.982708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.982688    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:29.987961 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.987924    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:48:29.988144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.988004    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:48:29.988144 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.988079    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:48:29.988699 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.988672    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:48:29.989653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:29.989627    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"]
Nov 02 18:48:30.169914 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.169814    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.170142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.169965    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.170142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.170038    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmt29\" (UniqueName: \"kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.270709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.270615    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.270709 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.270662    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.270951 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.270829    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmt29\" (UniqueName: \"kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.271358 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.271337    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.273289 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.273268    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.279543 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.279514    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmt29\" (UniqueName: \"kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29\") pod \"maas-api-revoke-sub-e2fc9c5b-d6n4g\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.300508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.300451    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:30.587421 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:30.587394    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"]
Nov 02 18:48:30.590169 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:48:30.590138    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7251c1e5_f22c_4d78_989d_e50bf4fb6c4f.slice/crio-fdeda8d0c116d4dfdea02bd008ae38da0500b6ba371bc38bd366992e916114ec WatchSource:0}: Error finding container fdeda8d0c116d4dfdea02bd008ae38da0500b6ba371bc38bd366992e916114ec: Status 404 returned error can't find the container with id fdeda8d0c116d4dfdea02bd008ae38da0500b6ba371bc38bd366992e916114ec
Nov 02 18:48:31.360115 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:31.360073    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g" event={"ID":"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f","Type":"ContainerStarted","Data":"15451a7824f6c55dbf00a0ccd5414477fce70c8cbe7623bed00c462a998be467"}
Nov 02 18:48:31.360539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:31.360142    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g" event={"ID":"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f","Type":"ContainerStarted","Data":"fdeda8d0c116d4dfdea02bd008ae38da0500b6ba371bc38bd366992e916114ec"}
Nov 02 18:48:31.376658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:31.376606    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g" podStartSLOduration=2.376585457 podStartE2EDuration="2.376585457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:48:31.375123015 +0000 UTC m=+4523.954532401" watchObservedRunningTime="2026-11-02 18:48:31.376585457 +0000 UTC m=+4523.955994844"
Nov 02 18:48:32.368089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:32.368055    5042 generic.go:356] "Generic (PLEG): container finished" podID="7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" containerID="15451a7824f6c55dbf00a0ccd5414477fce70c8cbe7623bed00c462a998be467" exitCode=0
Nov 02 18:48:32.368089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:32.368094    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g" event={"ID":"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f","Type":"ContainerDied","Data":"15451a7824f6c55dbf00a0ccd5414477fce70c8cbe7623bed00c462a998be467"}
Nov 02 18:48:33.534506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.534474    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:33.706146 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.706106    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca\") pod \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") "
Nov 02 18:48:33.706341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.706192    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca\") pod \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") "
Nov 02 18:48:33.706341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.706227    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29\" (UniqueName: \"kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29\") pod \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\" (UID: \"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f\") "
Nov 02 18:48:33.706593 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.706567    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca" pod "7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" (UID: "7251c1e5-f22c-4d78-989d-e50bf4fb6c4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:33.708563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.708540    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca" pod "7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" (UID: "7251c1e5-f22c-4d78-989d-e50bf4fb6c4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:33.708672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.708659    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29" pod "7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" (UID: "7251c1e5-f22c-4d78-989d-e50bf4fb6c4f"). InnerVolumeSpecName "kube-api-access-lmt29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:33.807365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.807324    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:33.807365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.807359    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:33.807365 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:33.807369    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmt29\" (UniqueName: \"kubernetes.io/projected/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f-kube-api-access-lmt29\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:34.245671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:34.245631    5042 scope.go:122] "RemoveContainer" containerID="29b3d05741748637b5c7138a93347b4e69ee42e397610b34a198203e140398e4"
Nov 02 18:48:34.255680 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:34.255655    5042 scope.go:122] "RemoveContainer" containerID="ff4026e3a543bef4be977ca21ac76a5fae6503d2107888452453cdf29a0ec812"
Nov 02 18:48:34.381763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:34.381725    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"
Nov 02 18:48:34.381763 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:34.381744    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g" event={"ID":"7251c1e5-f22c-4d78-989d-e50bf4fb6c4f","Type":"ContainerDied","Data":"fdeda8d0c116d4dfdea02bd008ae38da0500b6ba371bc38bd366992e916114ec"}
Nov 02 18:48:34.382007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:34.381787    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdeda8d0c116d4dfdea02bd008ae38da0500b6ba371bc38bd366992e916114ec"
Nov 02 18:48:40.184590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.184538    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"]
Nov 02 18:48:40.185145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.185041    5042 cpu_manager.go:414] "containerMap: removing container" podUID="7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" containerName="revoke-keys"
Nov 02 18:48:40.185145 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.185054    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" containerName="revoke-keys"
Nov 02 18:48:40.185270 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.185160    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" containerName="revoke-keys"
Nov 02 18:48:40.190255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.190223    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.194469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.194440    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:48:40.195641 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.195606    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"]
Nov 02 18:48:40.196017 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.195788    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:48:40.196181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.195872    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:48:40.196181 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.195941    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:48:40.268738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.268706    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.268963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.268810    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmh6n\" (UniqueName: \"kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.268963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.268896    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.369560 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.369520    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.369785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.369634    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.369785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.369676    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmh6n\" (UniqueName: \"kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.370227 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.370204    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.372341 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.372316    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.378248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.378224    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmh6n\" (UniqueName: \"kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n\") pod \"maas-api-revoke-sub-5a29efb3-z7s8z\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.511072 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.511032    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:40.794691 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:40.794653    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"]
Nov 02 18:48:40.796827 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:48:40.796793    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5be8a0_3611_490c_aefb_c888d65f5819.slice/crio-3e6b787c2265406090cfffad997af4fcf47749b49b043187677c33209229f84c WatchSource:0}: Error finding container 3e6b787c2265406090cfffad997af4fcf47749b49b043187677c33209229f84c: Status 404 returned error can't find the container with id 3e6b787c2265406090cfffad997af4fcf47749b49b043187677c33209229f84c
Nov 02 18:48:41.429801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:41.429692    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z" event={"ID":"9d5be8a0-3611-490c-aefb-c888d65f5819","Type":"ContainerStarted","Data":"2a7079b7100cccbf2d43cd7834bf61a13067597a6c7c3f3b955e62a8f184ed41"}
Nov 02 18:48:41.430374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:41.429804    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z" event={"ID":"9d5be8a0-3611-490c-aefb-c888d65f5819","Type":"ContainerStarted","Data":"3e6b787c2265406090cfffad997af4fcf47749b49b043187677c33209229f84c"}
Nov 02 18:48:41.450413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:41.450348    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z" podStartSLOduration=1.450328456 podStartE2EDuration="1.450328456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:48:41.44582844 +0000 UTC m=+4534.025237829" watchObservedRunningTime="2026-11-02 18:48:41.450328456 +0000 UTC m=+4534.029737844"
Nov 02 18:48:42.463940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:42.463651    5042 generic.go:356] "Generic (PLEG): container finished" podID="9d5be8a0-3611-490c-aefb-c888d65f5819" containerID="2a7079b7100cccbf2d43cd7834bf61a13067597a6c7c3f3b955e62a8f184ed41" exitCode=0
Nov 02 18:48:42.463940 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:42.463909    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z" event={"ID":"9d5be8a0-3611-490c-aefb-c888d65f5819","Type":"ContainerDied","Data":"2a7079b7100cccbf2d43cd7834bf61a13067597a6c7c3f3b955e62a8f184ed41"}
Nov 02 18:48:44.005032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.004892    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:44.082428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.082331    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca\") pod \"9d5be8a0-3611-490c-aefb-c888d65f5819\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") "
Nov 02 18:48:44.082428 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.082386    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n\" (UniqueName: \"kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n\") pod \"9d5be8a0-3611-490c-aefb-c888d65f5819\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") "
Nov 02 18:48:44.082651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.082429    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca\") pod \"9d5be8a0-3611-490c-aefb-c888d65f5819\" (UID: \"9d5be8a0-3611-490c-aefb-c888d65f5819\") "
Nov 02 18:48:44.082875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.082807    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca" pod "9d5be8a0-3611-490c-aefb-c888d65f5819" (UID: "9d5be8a0-3611-490c-aefb-c888d65f5819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:44.083005 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.082957    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:44.084943 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.084915    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n" pod "9d5be8a0-3611-490c-aefb-c888d65f5819" (UID: "9d5be8a0-3611-490c-aefb-c888d65f5819"). InnerVolumeSpecName "kube-api-access-dmh6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:44.085051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.084923    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca" pod "9d5be8a0-3611-490c-aefb-c888d65f5819" (UID: "9d5be8a0-3611-490c-aefb-c888d65f5819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:44.184540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.184487    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmh6n\" (UniqueName: \"kubernetes.io/projected/9d5be8a0-3611-490c-aefb-c888d65f5819-kube-api-access-dmh6n\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:44.184540 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.184524    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5be8a0-3611-490c-aefb-c888d65f5819-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:44.529872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.529815    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"
Nov 02 18:48:44.529872 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.529858    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z" event={"ID":"9d5be8a0-3611-490c-aefb-c888d65f5819","Type":"ContainerDied","Data":"3e6b787c2265406090cfffad997af4fcf47749b49b043187677c33209229f84c"}
Nov 02 18:48:44.530159 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:44.529910    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6b787c2265406090cfffad997af4fcf47749b49b043187677c33209229f84c"
Nov 02 18:48:48.539630 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:48:48.539555    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:48:55.577359 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.577306    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"]
Nov 02 18:48:55.581304 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.581277    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nk9tk"]
Nov 02 18:48:55.588269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.588238    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"]
Nov 02 18:48:55.588815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.588794    5042 cpu_manager.go:414] "containerMap: removing container" podUID="9d5be8a0-3611-490c-aefb-c888d65f5819" containerName="revoke-keys"
Nov 02 18:48:55.588815 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.588813    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5be8a0-3611-490c-aefb-c888d65f5819" containerName="revoke-keys"
Nov 02 18:48:55.589022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.588919    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5be8a0-3611-490c-aefb-c888d65f5819" containerName="revoke-keys"
Nov 02 18:48:55.591620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.591600    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.595780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.595550    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:48:55.595780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.595644    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:48:55.599852 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.598096    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:48:55.601051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.598532    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:48:55.601051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.601015    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"]
Nov 02 18:48:55.710925 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.710822    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2pt\" (UniqueName: \"kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.711112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.711021    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.711112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.711049    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.811712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.811670    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2pt\" (UniqueName: \"kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.811987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.811765    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.811987 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.811792    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.812508 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.812481    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.814482 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.814455    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.820051 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.820012    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2pt\" (UniqueName: \"kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt\") pod \"maas-api-revoke-sub-3d4156f1-9fd5q\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:55.918614 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:55.918520    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:56.197886 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:56.197855    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"]
Nov 02 18:48:56.199742 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:48:56.199716    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648aad58_d68a_4e02_9dac_83506a547af4.slice/crio-c7ef53608a39d424d38c7c2448b76d5a12d2d91983646624d2835b2add20905c WatchSource:0}: Error finding container c7ef53608a39d424d38c7c2448b76d5a12d2d91983646624d2835b2add20905c: Status 404 returned error can't find the container with id c7ef53608a39d424d38c7c2448b76d5a12d2d91983646624d2835b2add20905c
Nov 02 18:48:56.606595 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:56.606555    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q" event={"ID":"648aad58-d68a-4e02-9dac-83506a547af4","Type":"ContainerStarted","Data":"08a020fe98adaea71bd2717c59b7e9f21067fe3c959df23e0c109e74a9ccd632"}
Nov 02 18:48:56.607163 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:56.606617    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q" event={"ID":"648aad58-d68a-4e02-9dac-83506a547af4","Type":"ContainerStarted","Data":"c7ef53608a39d424d38c7c2448b76d5a12d2d91983646624d2835b2add20905c"}
Nov 02 18:48:56.623710 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:56.623653    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q" podStartSLOduration=1.623638068 podStartE2EDuration="1.623638068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:48:56.62136825 +0000 UTC m=+4549.200777638" watchObservedRunningTime="2026-11-02 18:48:56.623638068 +0000 UTC m=+4549.203047454"
Nov 02 18:48:57.542269 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:57.542230    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636a850c-6e08-4558-a9ee-409e993dc075" path="/var/lib/kubelet/pods/636a850c-6e08-4558-a9ee-409e993dc075/volumes"
Nov 02 18:48:57.614273 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:57.614241    5042 generic.go:356] "Generic (PLEG): container finished" podID="648aad58-d68a-4e02-9dac-83506a547af4" containerID="08a020fe98adaea71bd2717c59b7e9f21067fe3c959df23e0c109e74a9ccd632" exitCode=0
Nov 02 18:48:57.614694 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:57.614328    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q" event={"ID":"648aad58-d68a-4e02-9dac-83506a547af4","Type":"ContainerDied","Data":"08a020fe98adaea71bd2717c59b7e9f21067fe3c959df23e0c109e74a9ccd632"}
Nov 02 18:48:58.766261 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.766231    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:58.845698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.845660    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca\") pod \"648aad58-d68a-4e02-9dac-83506a547af4\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") "
Nov 02 18:48:58.845928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.845710    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca\") pod \"648aad58-d68a-4e02-9dac-83506a547af4\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") "
Nov 02 18:48:58.845928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.845784    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt\" (UniqueName: \"kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt\") pod \"648aad58-d68a-4e02-9dac-83506a547af4\" (UID: \"648aad58-d68a-4e02-9dac-83506a547af4\") "
Nov 02 18:48:58.846142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.846102    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca" pod "648aad58-d68a-4e02-9dac-83506a547af4" (UID: "648aad58-d68a-4e02-9dac-83506a547af4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:48:58.848202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.848176    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca" pod "648aad58-d68a-4e02-9dac-83506a547af4" (UID: "648aad58-d68a-4e02-9dac-83506a547af4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:48:58.848705 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.848685    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt" pod "648aad58-d68a-4e02-9dac-83506a547af4" (UID: "648aad58-d68a-4e02-9dac-83506a547af4"). InnerVolumeSpecName "kube-api-access-nk2pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:48:58.946932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.946892    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:58.946932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.946930    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648aad58-d68a-4e02-9dac-83506a547af4-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:58.946932 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:58.946946    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk2pt\" (UniqueName: \"kubernetes.io/projected/648aad58-d68a-4e02-9dac-83506a547af4-kube-api-access-nk2pt\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:48:59.628009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:59.627980    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"
Nov 02 18:48:59.628215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:59.627957    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q" event={"ID":"648aad58-d68a-4e02-9dac-83506a547af4","Type":"ContainerDied","Data":"c7ef53608a39d424d38c7c2448b76d5a12d2d91983646624d2835b2add20905c"}
Nov 02 18:48:59.628215 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:48:59.628109    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ef53608a39d424d38c7c2448b76d5a12d2d91983646624d2835b2add20905c"
Nov 02 18:49:08.643067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:49:08.643029    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:49:08.643646 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:49:08.643605    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:49:18.539814 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:49:18.539776    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:49:34.285185 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:49:34.285153    5042 scope.go:122] "RemoveContainer" containerID="7445492808e4b6fbcbe1f3e60d6b04f98a241d807ee09fdbe0d521c134bdc428"
Nov 02 18:49:48.539542 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:49:48.539426    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:50:01.164765 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.164716    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"]
Nov 02 18:50:01.165423 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.165400    5042 cpu_manager.go:414] "containerMap: removing container" podUID="648aad58-d68a-4e02-9dac-83506a547af4" containerName="revoke-keys"
Nov 02 18:50:01.165521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.165426    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648aad58-d68a-4e02-9dac-83506a547af4" containerName="revoke-keys"
Nov 02 18:50:01.165599 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.165586    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="648aad58-d68a-4e02-9dac-83506a547af4" containerName="revoke-keys"
Nov 02 18:50:01.168903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.168881    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.173631 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.173606    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:50:01.173889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.173862    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:50:01.174062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.174045    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:50:01.174712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.174673    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:50:01.176562 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.176532    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"]
Nov 02 18:50:01.213892 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.213818    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.214071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.214038    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.214136 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.214082    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v7vn\" (UniqueName: \"kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.315264 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.315223    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.315479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.315315    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.315479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.315341    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v7vn\" (UniqueName: \"kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.315959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.315933    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.318290 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.318263    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.324212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.324186    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v7vn\" (UniqueName: \"kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn\") pod \"maas-api-revoke-sub-9ace16cc-rmcjx\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.487061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.487022    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:01.763975 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:01.763946    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"]
Nov 02 18:50:01.765454 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:50:01.765421    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605a9cd0_5e77_4b4d_bc36_07f4ffe014e8.slice/crio-b3b323727c38eb4870ae5e38c7854f3abc7e0c99695e54bb1fd29937ff183a34 WatchSource:0}: Error finding container b3b323727c38eb4870ae5e38c7854f3abc7e0c99695e54bb1fd29937ff183a34: Status 404 returned error can't find the container with id b3b323727c38eb4870ae5e38c7854f3abc7e0c99695e54bb1fd29937ff183a34
Nov 02 18:50:02.029551 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:02.029458    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx" event={"ID":"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8","Type":"ContainerStarted","Data":"b3b323727c38eb4870ae5e38c7854f3abc7e0c99695e54bb1fd29937ff183a34"}
Nov 02 18:50:03.036819 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:03.036788    5042 generic.go:356] "Generic (PLEG): container finished" podID="605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" containerID="fa7651e40336cef554331ddad69cb5028d92f641cf35b1eea55db3d576eed8e2" exitCode=0
Nov 02 18:50:03.037276 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:03.036879    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx" event={"ID":"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8","Type":"ContainerDied","Data":"fa7651e40336cef554331ddad69cb5028d92f641cf35b1eea55db3d576eed8e2"}
Nov 02 18:50:04.198617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.198591    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:04.247076 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.247043    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca\") pod \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") "
Nov 02 18:50:04.247283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.247127    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca\") pod \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") "
Nov 02 18:50:04.247283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.247149    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn\" (UniqueName: \"kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn\") pod \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\" (UID: \"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8\") "
Nov 02 18:50:04.247559 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.247522    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca" pod "605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" (UID: "605a9cd0-5e77-4b4d-bc36-07f4ffe014e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:50:04.249465 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.249433    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca" pod "605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" (UID: "605a9cd0-5e77-4b4d-bc36-07f4ffe014e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:50:04.249622 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.249601    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn" pod "605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" (UID: "605a9cd0-5e77-4b4d-bc36-07f4ffe014e8"). InnerVolumeSpecName "kube-api-access-7v7vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:50:04.348556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.348470    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:50:04.348556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.348504    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:50:04.348556 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:04.348514    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v7vn\" (UniqueName: \"kubernetes.io/projected/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8-kube-api-access-7v7vn\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:50:05.051521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:05.051488    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"
Nov 02 18:50:05.051751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:05.051516    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx" event={"ID":"605a9cd0-5e77-4b4d-bc36-07f4ffe014e8","Type":"ContainerDied","Data":"b3b323727c38eb4870ae5e38c7854f3abc7e0c99695e54bb1fd29937ff183a34"}
Nov 02 18:50:05.051751 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:05.051567    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b323727c38eb4870ae5e38c7854f3abc7e0c99695e54bb1fd29937ff183a34"
Nov 02 18:50:18.539675 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:50:18.539573    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:50:37.023212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:37.023178    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"]
Nov 02 18:50:37.026010 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:37.025981    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vtljg"]
Nov 02 18:50:37.543336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:37.543300    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f0d812-72f4-488d-8487-72037df581eb" path="/var/lib/kubelet/pods/31f0d812-72f4-488d-8487-72037df581eb/volumes"
Nov 02 18:50:48.539053 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:50:48.539011    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:50:53.030110 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:53.030068    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"]
Nov 02 18:50:53.033507 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:53.033475    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-hqzzx"]
Nov 02 18:50:53.544319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:53.544284    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2ffeff-aac2-4bfa-bae7-fa524cc9904d" path="/var/lib/kubelet/pods/cd2ffeff-aac2-4bfa-bae7-fa524cc9904d/volumes"
Nov 02 18:50:58.311148 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.311109    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"]
Nov 02 18:50:58.311608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.311555    5042 cpu_manager.go:414] "containerMap: removing container" podUID="605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" containerName="revoke-keys"
Nov 02 18:50:58.311608 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.311567    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" containerName="revoke-keys"
Nov 02 18:50:58.311741 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.311651    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" containerName="revoke-keys"
Nov 02 18:50:58.314919 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.314901    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.320758 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.320731    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:50:58.320964 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.320855    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:50:58.321057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.321036    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:50:58.321189 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.321172    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:50:58.324985 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.324961    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"]
Nov 02 18:50:58.410625 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.410581    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.410877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.410750    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.410877 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.410782    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpx69\" (UniqueName: \"kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.512400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.512360    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.512400 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.512403    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpx69\" (UniqueName: \"kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.512660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.512463    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.513177 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.513152    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.515074 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.515051    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.520351 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.520323    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpx69\" (UniqueName: \"kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.632735 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.632639    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:50:58.915476 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:58.915443    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"]
Nov 02 18:50:58.918001 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:50:58.917955    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf80e7e_5c61_4327_b5c6_45df4063dcbc.slice/crio-31f751468ac3074bb930d9c5bb93900817a7cec04648bd4bba8c0efae8d40ccf WatchSource:0}: Error finding container 31f751468ac3074bb930d9c5bb93900817a7cec04648bd4bba8c0efae8d40ccf: Status 404 returned error can't find the container with id 31f751468ac3074bb930d9c5bb93900817a7cec04648bd4bba8c0efae8d40ccf
Nov 02 18:50:59.407874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:59.407821    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl" event={"ID":"bcf80e7e-5c61-4327-b5c6-45df4063dcbc","Type":"ContainerStarted","Data":"f59ef923dc470a45bdfe39b577e10120e093b751315e6b3f59a5cfd41b9104ce"}
Nov 02 18:50:59.408312 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:59.407886    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl" event={"ID":"bcf80e7e-5c61-4327-b5c6-45df4063dcbc","Type":"ContainerStarted","Data":"31f751468ac3074bb930d9c5bb93900817a7cec04648bd4bba8c0efae8d40ccf"}
Nov 02 18:50:59.422893 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:50:59.422565    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl" podStartSLOduration=1.422547313 podStartE2EDuration="1.422547313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:59.422360613 +0000 UTC m=+4672.001770022" watchObservedRunningTime="2026-11-02 18:50:59.422547313 +0000 UTC m=+4672.001956699"
Nov 02 18:51:00.415888 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:00.415831    5042 generic.go:356] "Generic (PLEG): container finished" podID="bcf80e7e-5c61-4327-b5c6-45df4063dcbc" containerID="f59ef923dc470a45bdfe39b577e10120e093b751315e6b3f59a5cfd41b9104ce" exitCode=0
Nov 02 18:51:00.416315 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:00.415890    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl" event={"ID":"bcf80e7e-5c61-4327-b5c6-45df4063dcbc","Type":"ContainerDied","Data":"f59ef923dc470a45bdfe39b577e10120e093b751315e6b3f59a5cfd41b9104ce"}
Nov 02 18:51:01.571183 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.571157    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:51:01.746237 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.746200    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca\") pod \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") "
Nov 02 18:51:01.746473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.746320    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca\") pod \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") "
Nov 02 18:51:01.746473 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.746375    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69\" (UniqueName: \"kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69\") pod \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\" (UID: \"bcf80e7e-5c61-4327-b5c6-45df4063dcbc\") "
Nov 02 18:51:01.746744 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.746721    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca" pod "bcf80e7e-5c61-4327-b5c6-45df4063dcbc" (UID: "bcf80e7e-5c61-4327-b5c6-45df4063dcbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:51:01.748607 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.748575    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca" pod "bcf80e7e-5c61-4327-b5c6-45df4063dcbc" (UID: "bcf80e7e-5c61-4327-b5c6-45df4063dcbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:51:01.748745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.748724    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69" pod "bcf80e7e-5c61-4327-b5c6-45df4063dcbc" (UID: "bcf80e7e-5c61-4327-b5c6-45df4063dcbc"). InnerVolumeSpecName "kube-api-access-bpx69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:01.847366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.847327    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:01.847366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.847357    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:01.847366 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:01.847369    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpx69\" (UniqueName: \"kubernetes.io/projected/bcf80e7e-5c61-4327-b5c6-45df4063dcbc-kube-api-access-bpx69\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:02.428723 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:02.428686    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"
Nov 02 18:51:02.428945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:02.428722    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl" event={"ID":"bcf80e7e-5c61-4327-b5c6-45df4063dcbc","Type":"ContainerDied","Data":"31f751468ac3074bb930d9c5bb93900817a7cec04648bd4bba8c0efae8d40ccf"}
Nov 02 18:51:02.428945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:02.428804    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31f751468ac3074bb930d9c5bb93900817a7cec04648bd4bba8c0efae8d40ccf"
Nov 02 18:51:04.021889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:04.021850    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"]
Nov 02 18:51:04.024411 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:04.024381    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-szlvq"]
Nov 02 18:51:05.542162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:05.542128    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb6ba59-bffc-45d5-b346-56c458102f03" path="/var/lib/kubelet/pods/9cb6ba59-bffc-45d5-b346-56c458102f03/volumes"
Nov 02 18:51:09.321712 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:09.321670    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"]
Nov 02 18:51:09.324620 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:09.324597    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3bd024-2qztl"]
Nov 02 18:51:09.544827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:09.544772    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf80e7e-5c61-4327-b5c6-45df4063dcbc" path="/var/lib/kubelet/pods/bcf80e7e-5c61-4327-b5c6-45df4063dcbc/volumes"
Nov 02 18:51:10.943122 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.943089    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"]
Nov 02 18:51:10.945731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.945702    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9fd5q"]
Nov 02 18:51:10.967630 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.967596    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"]
Nov 02 18:51:10.968070 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.968057    5042 cpu_manager.go:414] "containerMap: removing container" podUID="bcf80e7e-5c61-4327-b5c6-45df4063dcbc" containerName="revoke-keys"
Nov 02 18:51:10.968139 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.968072    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf80e7e-5c61-4327-b5c6-45df4063dcbc" containerName="revoke-keys"
Nov 02 18:51:10.968179 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.968156    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf80e7e-5c61-4327-b5c6-45df4063dcbc" containerName="revoke-keys"
Nov 02 18:51:10.972757 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.972733    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:10.978446 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.978406    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:51:10.978601 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.978485    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:51:10.981514 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.981473    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"]
Nov 02 18:51:10.994469 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.994440    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:51:10.994633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:10.994569    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:51:11.043698 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.043665    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfvh8\" (UniqueName: \"kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.043927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.043768    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.043927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.043866    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.144506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.144465    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfvh8\" (UniqueName: \"kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.144701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.144539    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.144701 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.144571    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.145225 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.145201    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.147295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.147275    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.152475 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.152448    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfvh8\" (UniqueName: \"kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8\") pod \"maas-api-revoke-sub-3d4156f1-xs8gl\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.297745 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.297648    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:11.542413 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.542368    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648aad58-d68a-4e02-9dac-83506a547af4" path="/var/lib/kubelet/pods/648aad58-d68a-4e02-9dac-83506a547af4/volumes"
Nov 02 18:51:11.579019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:11.578984    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"]
Nov 02 18:51:11.580447 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:51:11.580415    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f348999_4af2_4c94_81fd_6ee0f7384246.slice/crio-4f3c885398b2cea3396ff5f25ccc0152ae294cfe444cd43f042c328e884b10d2 WatchSource:0}: Error finding container 4f3c885398b2cea3396ff5f25ccc0152ae294cfe444cd43f042c328e884b10d2: Status 404 returned error can't find the container with id 4f3c885398b2cea3396ff5f25ccc0152ae294cfe444cd43f042c328e884b10d2
Nov 02 18:51:12.494783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:12.494750    5042 generic.go:356] "Generic (PLEG): container finished" podID="6f348999-4af2-4c94-81fd-6ee0f7384246" containerID="5bb50b1e09bbcf69c3ab5074c0e8a196a9ccf16b7bc15440bf067a4122314c50" exitCode=0
Nov 02 18:51:12.495187 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:12.494833    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl" event={"ID":"6f348999-4af2-4c94-81fd-6ee0f7384246","Type":"ContainerDied","Data":"5bb50b1e09bbcf69c3ab5074c0e8a196a9ccf16b7bc15440bf067a4122314c50"}
Nov 02 18:51:12.495187 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:12.494914    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl" event={"ID":"6f348999-4af2-4c94-81fd-6ee0f7384246","Type":"ContainerStarted","Data":"4f3c885398b2cea3396ff5f25ccc0152ae294cfe444cd43f042c328e884b10d2"}
Nov 02 18:51:13.650506 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.650476    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:13.672555 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.672519    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8\" (UniqueName: \"kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8\") pod \"6f348999-4af2-4c94-81fd-6ee0f7384246\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") "
Nov 02 18:51:13.672748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.672608    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca\") pod \"6f348999-4af2-4c94-81fd-6ee0f7384246\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") "
Nov 02 18:51:13.672748 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.672735    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca\") pod \"6f348999-4af2-4c94-81fd-6ee0f7384246\" (UID: \"6f348999-4af2-4c94-81fd-6ee0f7384246\") "
Nov 02 18:51:13.673016 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.672985    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca" pod "6f348999-4af2-4c94-81fd-6ee0f7384246" (UID: "6f348999-4af2-4c94-81fd-6ee0f7384246"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:51:13.673162 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.673112    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:13.675202 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.675173    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca" pod "6f348999-4af2-4c94-81fd-6ee0f7384246" (UID: "6f348999-4af2-4c94-81fd-6ee0f7384246"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:51:13.675565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.675535    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8" pod "6f348999-4af2-4c94-81fd-6ee0f7384246" (UID: "6f348999-4af2-4c94-81fd-6ee0f7384246"). InnerVolumeSpecName "kube-api-access-mfvh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:13.774355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.774255    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfvh8\" (UniqueName: \"kubernetes.io/projected/6f348999-4af2-4c94-81fd-6ee0f7384246-kube-api-access-mfvh8\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:13.774355 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:13.774289    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f348999-4af2-4c94-81fd-6ee0f7384246-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:14.514831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:14.514796    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"
Nov 02 18:51:14.515058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:14.514802    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl" event={"ID":"6f348999-4af2-4c94-81fd-6ee0f7384246","Type":"ContainerDied","Data":"4f3c885398b2cea3396ff5f25ccc0152ae294cfe444cd43f042c328e884b10d2"}
Nov 02 18:51:14.515058 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:14.514922    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f3c885398b2cea3396ff5f25ccc0152ae294cfe444cd43f042c328e884b10d2"
Nov 02 18:51:18.543979 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:51:18.543935    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:51:23.641920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:23.641007    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:51:23.641920 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:23.641081    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:51:26.404238 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:26.404201    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"]
Nov 02 18:51:26.404731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:26.404445    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" podUID="588a37ce-4b38-4224-9f67-adb7273f76b5" containerName="manager" containerID="cri-o://f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4" gracePeriod=10
Nov 02 18:51:27.030441 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.030413    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"]
Nov 02 18:51:27.033404 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.033377    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-zxqqs"]
Nov 02 18:51:27.204307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.204276    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 18:51:27.296437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.296332    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume\") pod \"588a37ce-4b38-4224-9f67-adb7273f76b5\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") "
Nov 02 18:51:27.296594 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.296442    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp\" (UniqueName: \"kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp\") pod \"588a37ce-4b38-4224-9f67-adb7273f76b5\" (UID: \"588a37ce-4b38-4224-9f67-adb7273f76b5\") "
Nov 02 18:51:27.300220 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.300185    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume" pod "588a37ce-4b38-4224-9f67-adb7273f76b5" (UID: "588a37ce-4b38-4224-9f67-adb7273f76b5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:51:27.301201 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.301169    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp" pod "588a37ce-4b38-4224-9f67-adb7273f76b5" (UID: "588a37ce-4b38-4224-9f67-adb7273f76b5"). InnerVolumeSpecName "kube-api-access-npscp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:51:27.397716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.397672    5042 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588a37ce-4b38-4224-9f67-adb7273f76b5-extensions-socket-volume\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:27.397716 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.397709    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npscp\" (UniqueName: \"kubernetes.io/projected/588a37ce-4b38-4224-9f67-adb7273f76b5-kube-api-access-npscp\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:51:27.544031 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.543996    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36c21f17-2083-4b0d-a1fa-30b67d26214f" path="/var/lib/kubelet/pods/36c21f17-2083-4b0d-a1fa-30b67d26214f/volumes"
Nov 02 18:51:27.607371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.607336    5042 generic.go:356] "Generic (PLEG): container finished" podID="588a37ce-4b38-4224-9f67-adb7273f76b5" containerID="f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4" exitCode=0
Nov 02 18:51:27.607584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.607382    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"
Nov 02 18:51:27.607584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.607387    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" event={"ID":"588a37ce-4b38-4224-9f67-adb7273f76b5","Type":"ContainerDied","Data":"f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4"}
Nov 02 18:51:27.607584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.607527    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl" event={"ID":"588a37ce-4b38-4224-9f67-adb7273f76b5","Type":"ContainerDied","Data":"f5f4e67eb6430282de17ae512983e8735f0103d0cabffcb48267ca6b5c50953e"}
Nov 02 18:51:27.607584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.607562    5042 scope.go:122] "RemoveContainer" containerID="f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4"
Nov 02 18:51:27.626384 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.626352    5042 scope.go:122] "RemoveContainer" containerID="f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4"
Nov 02 18:51:27.626756 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.626733    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"]
Nov 02 18:51:27.626756 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:51:27.626728    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4\": container with ID starting with f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4 not found: ID does not exist" containerID="f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4"
Nov 02 18:51:27.627055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.626805    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4"} err="failed to get container status \"f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4\": rpc error: code = NotFound desc = could not find container \"f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4\": container with ID starting with f1acf0523bf23066ad0d31fc41d1cf84b963452f74ceee5c4028ad1b9f49a9e4 not found: ID does not exist"
Nov 02 18:51:27.630534 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:27.630510    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f56vl"]
Nov 02 18:51:29.544988 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:29.544940    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588a37ce-4b38-4224-9f67-adb7273f76b5" path="/var/lib/kubelet/pods/588a37ce-4b38-4224-9f67-adb7273f76b5/volumes"
Nov 02 18:51:34.326426 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:34.326394    5042 scope.go:122] "RemoveContainer" containerID="72942d1e5e7d7f9d17a4b8a7ae9b8015ba0772e364fd77c3805fe89a828b607b"
Nov 02 18:51:37.024590 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:37.024553    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"]
Nov 02 18:51:37.026799 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:37.026781    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-nq558"]
Nov 02 18:51:37.542364 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:37.542328    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c223e0-4d82-4d7f-a872-cfa5568489c3" path="/var/lib/kubelet/pods/70c223e0-4d82-4d7f-a872-cfa5568489c3/volumes"
Nov 02 18:51:48.538873 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:51:48.538769    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:51:58.023826 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:58.023783    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"]
Nov 02 18:51:58.026055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:58.026031    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4qtxn"]
Nov 02 18:51:59.543060 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:51:59.543027    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1de37c1-3a4f-4aff-a71a-15188f2f8816" path="/var/lib/kubelet/pods/a1de37c1-3a4f-4aff-a71a-15188f2f8816/volumes"
Nov 02 18:52:14.023430 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:14.023373    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"]
Nov 02 18:52:14.025882 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:14.025858    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-67jxz"]
Nov 02 18:52:15.543363 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:15.543321    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cae6cf-f27e-4198-b65d-aa2e190e6d01" path="/var/lib/kubelet/pods/78cae6cf-f27e-4198-b65d-aa2e190e6d01/volumes"
Nov 02 18:52:18.540306 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:52:18.540267    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:52:27.516274 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.516236    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"]
Nov 02 18:52:27.516880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.516864    5042 cpu_manager.go:414] "containerMap: removing container" podUID="6f348999-4af2-4c94-81fd-6ee0f7384246" containerName="revoke-keys"
Nov 02 18:52:27.516935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.516881    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f348999-4af2-4c94-81fd-6ee0f7384246" containerName="revoke-keys"
Nov 02 18:52:27.516935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.516914    5042 cpu_manager.go:414] "containerMap: removing container" podUID="588a37ce-4b38-4224-9f67-adb7273f76b5" containerName="manager"
Nov 02 18:52:27.516935 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.516919    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588a37ce-4b38-4224-9f67-adb7273f76b5" containerName="manager"
Nov 02 18:52:27.517037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.517001    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f348999-4af2-4c94-81fd-6ee0f7384246" containerName="revoke-keys"
Nov 02 18:52:27.517037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.517017    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="588a37ce-4b38-4224-9f67-adb7273f76b5" containerName="manager"
Nov 02 18:52:27.520278 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.520261    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.524553 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.524528    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2v2nf\""
Nov 02 18:52:27.530634 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.530609    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"]
Nov 02 18:52:27.617394 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.617343    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmgw2\" (UniqueName: \"kubernetes.io/projected/b5e8eadd-5714-470d-9adb-999d78f52191-kube-api-access-xmgw2\") pod \"kuadrant-operator-controller-manager-678d87766f-8c4p4\" (UID: \"b5e8eadd-5714-470d-9adb-999d78f52191\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.617629 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.617602    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5e8eadd-5714-470d-9adb-999d78f52191-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8c4p4\" (UID: \"b5e8eadd-5714-470d-9adb-999d78f52191\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.718501 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.718455    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5e8eadd-5714-470d-9adb-999d78f52191-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8c4p4\" (UID: \"b5e8eadd-5714-470d-9adb-999d78f52191\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.718731 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.718560    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmgw2\" (UniqueName: \"kubernetes.io/projected/b5e8eadd-5714-470d-9adb-999d78f52191-kube-api-access-xmgw2\") pod \"kuadrant-operator-controller-manager-678d87766f-8c4p4\" (UID: \"b5e8eadd-5714-470d-9adb-999d78f52191\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.718875 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.718815    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b5e8eadd-5714-470d-9adb-999d78f52191-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8c4p4\" (UID: \"b5e8eadd-5714-470d-9adb-999d78f52191\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.732624 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.732589    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmgw2\" (UniqueName: \"kubernetes.io/projected/b5e8eadd-5714-470d-9adb-999d78f52191-kube-api-access-xmgw2\") pod \"kuadrant-operator-controller-manager-678d87766f-8c4p4\" (UID: \"b5e8eadd-5714-470d-9adb-999d78f52191\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:27.835172 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:27.835076    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:28.123034 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:28.123000    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"]
Nov 02 18:52:28.124722 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:52:28.124690    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e8eadd_5714_470d_9adb_999d78f52191.slice/crio-8a13246a2c6e5986b953ce24d0430c9b0a3c7a3a115a462b685d7d8047da7bae WatchSource:0}: Error finding container 8a13246a2c6e5986b953ce24d0430c9b0a3c7a3a115a462b685d7d8047da7bae: Status 404 returned error can't find the container with id 8a13246a2c6e5986b953ce24d0430c9b0a3c7a3a115a462b685d7d8047da7bae
Nov 02 18:52:28.994313 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:28.994274    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4" event={"ID":"b5e8eadd-5714-470d-9adb-999d78f52191","Type":"ContainerStarted","Data":"a3de7e1240f7cf7725402315fa6422e8385df19f0a1d13ff7f3e180867068892"}
Nov 02 18:52:28.994820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:28.994361    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4" event={"ID":"b5e8eadd-5714-470d-9adb-999d78f52191","Type":"ContainerStarted","Data":"8a13246a2c6e5986b953ce24d0430c9b0a3c7a3a115a462b685d7d8047da7bae"}
Nov 02 18:52:28.994820 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:28.994474    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:29.016539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:29.016490    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4" podStartSLOduration=2.016476141 podStartE2EDuration="2.016476141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:52:29.013612037 +0000 UTC m=+4761.593021618" watchObservedRunningTime="2026-11-02 18:52:29.016476141 +0000 UTC m=+4761.595885528"
Nov 02 18:52:29.028346 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:29.028313    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"]
Nov 02 18:52:29.032450 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:29.032420    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-f724k"]
Nov 02 18:52:29.542287 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:29.542256    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dd54e2-0b88-443f-bdf4-43c4b14a89da" path="/var/lib/kubelet/pods/31dd54e2-0b88-443f-bdf4-43c4b14a89da/volumes"
Nov 02 18:52:34.358971 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:34.358932    5042 scope.go:122] "RemoveContainer" containerID="77d22f0c963fd295695b1a392cf5ba7c02956a191b266ec726956115f7f56e9b"
Nov 02 18:52:34.368454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:34.368435    5042 scope.go:122] "RemoveContainer" containerID="14a529fd9c823f130b1f4d2cf7988e62f41144179830dcc2fd57336b2fd1693b"
Nov 02 18:52:34.376947 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:34.376920    5042 scope.go:122] "RemoveContainer" containerID="d2e4d93cdd198e546a96b0dead00790b47908ef391789224e13f5768edaa3157"
Nov 02 18:52:34.385307 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:34.385277    5042 scope.go:122] "RemoveContainer" containerID="e1d6f04683bd9a06f88670a753d984d467ccdd2cf5e819892028d3cc6182606e"
Nov 02 18:52:40.002565 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:40.002531    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8c4p4"
Nov 02 18:52:44.024660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:44.024614    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"]
Nov 02 18:52:44.027320 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:44.027285    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4cmpq"]
Nov 02 18:52:45.542578 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:45.542541    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458ad5d6-9ea1-471b-bbf3-e13056b7dc58" path="/var/lib/kubelet/pods/458ad5d6-9ea1-471b-bbf3-e13056b7dc58/volumes"
Nov 02 18:52:48.539321 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:52:48.539285    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:52:52.787934 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.787894    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"]
Nov 02 18:52:52.791672 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.791651    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:52.797190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.797159    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:52:52.797374 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.797301    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:52:52.797455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.797395    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:52:52.797455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.797439    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:52:52.799539 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.799499    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"]
Nov 02 18:52:52.913089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.913052    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:52.913089 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.913088    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsff\" (UniqueName: \"kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:52.913321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:52.913249    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.014663 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.014623    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.014926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.014713    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.014926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.014732    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsff\" (UniqueName: \"kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.015375 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.015348    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.017538 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.017513    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.022970 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.022948    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsff\" (UniqueName: \"kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff\") pod \"maas-api-revoke-sub-d576e68b-b8q5x\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.108037 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.107950    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:53.393923 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:53.393784    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"]
Nov 02 18:52:53.396535 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:52:53.396507    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547f8787_a024_4ce7_9ce1_b67d301e693b.slice/crio-08c1eac565e6a9558a405b3dbf86fe2d7bc46f477749880d7c79cb2ae189a0e5 WatchSource:0}: Error finding container 08c1eac565e6a9558a405b3dbf86fe2d7bc46f477749880d7c79cb2ae189a0e5: Status 404 returned error can't find the container with id 08c1eac565e6a9558a405b3dbf86fe2d7bc46f477749880d7c79cb2ae189a0e5
Nov 02 18:52:54.178749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:54.178708    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x" event={"ID":"547f8787-a024-4ce7-9ce1-b67d301e693b","Type":"ContainerStarted","Data":"15598b9f48079789199fecda7425330261c130aae7ff6ef66de479263ae8bc72"}
Nov 02 18:52:54.179190 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:54.178771    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x" event={"ID":"547f8787-a024-4ce7-9ce1-b67d301e693b","Type":"ContainerStarted","Data":"08c1eac565e6a9558a405b3dbf86fe2d7bc46f477749880d7c79cb2ae189a0e5"}
Nov 02 18:52:54.196686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:54.196633    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x" podStartSLOduration=2.196618809 podStartE2EDuration="2.196618809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:52:54.193785726 +0000 UTC m=+4786.773195115" watchObservedRunningTime="2026-11-02 18:52:54.196618809 +0000 UTC m=+4786.776028194"
Nov 02 18:52:55.185298 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:55.185265    5042 generic.go:356] "Generic (PLEG): container finished" podID="547f8787-a024-4ce7-9ce1-b67d301e693b" containerID="15598b9f48079789199fecda7425330261c130aae7ff6ef66de479263ae8bc72" exitCode=0
Nov 02 18:52:55.185738 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:55.185351    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x" event={"ID":"547f8787-a024-4ce7-9ce1-b67d301e693b","Type":"ContainerDied","Data":"15598b9f48079789199fecda7425330261c130aae7ff6ef66de479263ae8bc72"}
Nov 02 18:52:56.339105 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.339073    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:56.449471 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.449357    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca\") pod \"547f8787-a024-4ce7-9ce1-b67d301e693b\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") "
Nov 02 18:52:56.449651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.449499    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff\" (UniqueName: \"kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff\") pod \"547f8787-a024-4ce7-9ce1-b67d301e693b\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") "
Nov 02 18:52:56.449651 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.449547    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca\") pod \"547f8787-a024-4ce7-9ce1-b67d301e693b\" (UID: \"547f8787-a024-4ce7-9ce1-b67d301e693b\") "
Nov 02 18:52:56.449997 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.449965    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca" pod "547f8787-a024-4ce7-9ce1-b67d301e693b" (UID: "547f8787-a024-4ce7-9ce1-b67d301e693b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:52:56.451812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.451782    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff" pod "547f8787-a024-4ce7-9ce1-b67d301e693b" (UID: "547f8787-a024-4ce7-9ce1-b67d301e693b"). InnerVolumeSpecName "kube-api-access-zzsff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:52:56.451812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.451787    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca" pod "547f8787-a024-4ce7-9ce1-b67d301e693b" (UID: "547f8787-a024-4ce7-9ce1-b67d301e693b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:52:56.551085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.551039    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:52:56.551085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.551079    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/547f8787-a024-4ce7-9ce1-b67d301e693b-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:52:56.551085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:56.551093    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzsff\" (UniqueName: \"kubernetes.io/projected/547f8787-a024-4ce7-9ce1-b67d301e693b-kube-api-access-zzsff\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:52:57.199383 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:57.199356    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"
Nov 02 18:52:57.199573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:57.199403    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x" event={"ID":"547f8787-a024-4ce7-9ce1-b67d301e693b","Type":"ContainerDied","Data":"08c1eac565e6a9558a405b3dbf86fe2d7bc46f477749880d7c79cb2ae189a0e5"}
Nov 02 18:52:57.199573 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:52:57.199453    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c1eac565e6a9558a405b3dbf86fe2d7bc46f477749880d7c79cb2ae189a0e5"
Nov 02 18:53:08.989071 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:08.989031    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"]
Nov 02 18:53:08.990679 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:08.990647    5042 cpu_manager.go:414] "containerMap: removing container" podUID="547f8787-a024-4ce7-9ce1-b67d301e693b" containerName="revoke-keys"
Nov 02 18:53:08.990874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:08.990858    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547f8787-a024-4ce7-9ce1-b67d301e693b" containerName="revoke-keys"
Nov 02 18:53:08.991258 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:08.991242    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="547f8787-a024-4ce7-9ce1-b67d301e693b" containerName="revoke-keys"
Nov 02 18:53:09.002167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.002109    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"]
Nov 02 18:53:09.002512 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.002212    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.010454 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.010425    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:53:09.010617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.010528    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ntvjb\""
Nov 02 18:53:09.010617 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.010575    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:53:09.010740 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.010636    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:53:09.097884 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.097818    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qfh\" (UniqueName: \"kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.098055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.098008    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.098055 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.098048    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.199098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.199054    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.199098 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.199100    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.199345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.199151    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qfh\" (UniqueName: \"kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.199792 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.199771    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.201737 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.201717    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.210798 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.210754    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qfh\" (UniqueName: \"kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh\") pod \"maas-api-revoke-sub-d2bc93bc-xlcjr\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.323903 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.323800    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:09.605382 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:09.605354    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"]
Nov 02 18:53:09.607303 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:53:09.607271    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fce1258_3b3e_43b7_829d_6863fa402353.slice/crio-7a2cb6dfa81a48d608e3f9ddc789a0b4eea55dbf623a9d293469870cb285399b WatchSource:0}: Error finding container 7a2cb6dfa81a48d608e3f9ddc789a0b4eea55dbf623a9d293469870cb285399b: Status 404 returned error can't find the container with id 7a2cb6dfa81a48d608e3f9ddc789a0b4eea55dbf623a9d293469870cb285399b
Nov 02 18:53:10.050994 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.050964    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:53:10.053357 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.053335    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:53:10.053827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.053807    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:53:10.056645 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.056626    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:53:10.291681 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.291638    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr" event={"ID":"0fce1258-3b3e-43b7-829d-6863fa402353","Type":"ContainerStarted","Data":"f9e6c0cb725e59e296c439832baccd73af34c9c12482c6b9a2487a9e18efd125"}
Nov 02 18:53:10.291927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.291704    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr" event={"ID":"0fce1258-3b3e-43b7-829d-6863fa402353","Type":"ContainerStarted","Data":"7a2cb6dfa81a48d608e3f9ddc789a0b4eea55dbf623a9d293469870cb285399b"}
Nov 02 18:53:10.308812 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:10.308764    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr" podStartSLOduration=2.308748353 podStartE2EDuration="2.308748353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:10.307877976 +0000 UTC m=+4802.887287364" watchObservedRunningTime="2026-11-02 18:53:10.308748353 +0000 UTC m=+4802.888157732"
Nov 02 18:53:11.299749 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:11.299716    5042 generic.go:356] "Generic (PLEG): container finished" podID="0fce1258-3b3e-43b7-829d-6863fa402353" containerID="f9e6c0cb725e59e296c439832baccd73af34c9c12482c6b9a2487a9e18efd125" exitCode=0
Nov 02 18:53:11.300196 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:11.299808    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr" event={"ID":"0fce1258-3b3e-43b7-829d-6863fa402353","Type":"ContainerDied","Data":"f9e6c0cb725e59e296c439832baccd73af34c9c12482c6b9a2487a9e18efd125"}
Nov 02 18:53:12.460022 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.459996    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:12.539419 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.539385    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca\") pod \"0fce1258-3b3e-43b7-829d-6863fa402353\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") "
Nov 02 18:53:12.539597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.539442    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh\" (UniqueName: \"kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh\") pod \"0fce1258-3b3e-43b7-829d-6863fa402353\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") "
Nov 02 18:53:12.539665 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.539637    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca\") pod \"0fce1258-3b3e-43b7-829d-6863fa402353\" (UID: \"0fce1258-3b3e-43b7-829d-6863fa402353\") "
Nov 02 18:53:12.540009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.539981    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca" pod "0fce1258-3b3e-43b7-829d-6863fa402353" (UID: "0fce1258-3b3e-43b7-829d-6863fa402353"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:53:12.541690 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.541664    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca" pod "0fce1258-3b3e-43b7-829d-6863fa402353" (UID: "0fce1258-3b3e-43b7-829d-6863fa402353"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:53:12.541781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.541736    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh" pod "0fce1258-3b3e-43b7-829d-6863fa402353" (UID: "0fce1258-3b3e-43b7-829d-6863fa402353"). InnerVolumeSpecName "kube-api-access-p6qfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:53:12.641747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.641648    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-openshift-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:12.641930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.641766    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fce1258-3b3e-43b7-829d-6863fa402353-maas-api-xks-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:12.641930 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:12.641795    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qfh\" (UniqueName: \"kubernetes.io/projected/0fce1258-3b3e-43b7-829d-6863fa402353-kube-api-access-p6qfh\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:13.315713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:13.315676    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr" event={"ID":"0fce1258-3b3e-43b7-829d-6863fa402353","Type":"ContainerDied","Data":"7a2cb6dfa81a48d608e3f9ddc789a0b4eea55dbf623a9d293469870cb285399b"}
Nov 02 18:53:13.315713 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:13.315699    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"
Nov 02 18:53:13.316042 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:13.315733    5042 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2cb6dfa81a48d608e3f9ddc789a0b4eea55dbf623a9d293469870cb285399b"
Nov 02 18:53:18.538884 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:53:18.538771    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:53:21.039789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.039748    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"]
Nov 02 18:53:21.040253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.040211    5042 cpu_manager.go:414] "containerMap: removing container" podUID="0fce1258-3b3e-43b7-829d-6863fa402353" containerName="revoke-keys"
Nov 02 18:53:21.040253 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.040223    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fce1258-3b3e-43b7-829d-6863fa402353" containerName="revoke-keys"
Nov 02 18:53:21.040336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.040306    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fce1258-3b3e-43b7-829d-6863fa402353" containerName="revoke-keys"
Nov 02 18:53:21.043430 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.043409    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.054789 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.054752    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-tqgzn\""
Nov 02 18:53:21.055248 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.055223    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:53:21.055397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.055261    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 18:53:21.055397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.055259    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:53:21.055397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.055334    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:53:21.058336 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.058310    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"]
Nov 02 18:53:21.130885 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.130833    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.131085 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.130950    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.131154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.131089    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcrpd\" (UniqueName: \"kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.131154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.131139    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.232281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.232244    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.232281 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.232285    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.232521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.232352    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcrpd\" (UniqueName: \"kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.232521 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.232376    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.232954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.232929    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.235020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.234998    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.235134 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.235075    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.240207 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.240185    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcrpd\" (UniqueName: \"kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd\") pod \"maas-api-84f9c6994b-86mpt\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") " pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.367291 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.367206    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:21.645028 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.645002    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"]
Nov 02 18:53:21.647446 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:53:21.647419    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66490f7_e2bd_4e56_be55_e7390bc2ae5f.slice/crio-003dca1c79133e076d26e36f006255402e4f2d9576832979f43f9ceb1f75b721 WatchSource:0}: Error finding container 003dca1c79133e076d26e36f006255402e4f2d9576832979f43f9ceb1f75b721: Status 404 returned error can't find the container with id 003dca1c79133e076d26e36f006255402e4f2d9576832979f43f9ceb1f75b721
Nov 02 18:53:21.648788 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:21.648770    5042 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:53:22.372131 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:22.372092    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt" event={"ID":"b66490f7-e2bd-4e56-be55-e7390bc2ae5f","Type":"ContainerStarted","Data":"34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0"}
Nov 02 18:53:22.372557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:22.372156    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt" event={"ID":"b66490f7-e2bd-4e56-be55-e7390bc2ae5f","Type":"ContainerStarted","Data":"003dca1c79133e076d26e36f006255402e4f2d9576832979f43f9ceb1f75b721"}
Nov 02 18:53:22.372557 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:22.372183    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:22.389547 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:22.389498    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt" podStartSLOduration=0.847485305 podStartE2EDuration="1.389483005s" totalImagesPullingTime="541.9977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:22.387943776 +0000 UTC m=+4814.967353164" watchObservedRunningTime="2026-11-02 18:53:22.389483005 +0000 UTC m=+4814.968892433"
Nov 02 18:53:28.383425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.383394    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:28.535262 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.535225    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"]
Nov 02 18:53:28.539492 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.539460    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.547522 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.547214    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"]
Nov 02 18:53:28.712167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.712127    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1171b75e-3943-4a7d-a9b5-f4dea185ef43-maas-api-tls\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.712354 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.712254    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1171b75e-3943-4a7d-a9b5-f4dea185ef43-openshift-service-ca\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.712399 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.712359    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1171b75e-3943-4a7d-a9b5-f4dea185ef43-metrics-certs\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.712497 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.712465    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9jzw\" (UniqueName: \"kubernetes.io/projected/1171b75e-3943-4a7d-a9b5-f4dea185ef43-kube-api-access-v9jzw\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.813437 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.813394    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1171b75e-3943-4a7d-a9b5-f4dea185ef43-metrics-certs\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.813669 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.813464    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9jzw\" (UniqueName: \"kubernetes.io/projected/1171b75e-3943-4a7d-a9b5-f4dea185ef43-kube-api-access-v9jzw\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.813669 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.813561    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1171b75e-3943-4a7d-a9b5-f4dea185ef43-maas-api-tls\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.813669 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.813581    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1171b75e-3943-4a7d-a9b5-f4dea185ef43-openshift-service-ca\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.814319 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.814300    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1171b75e-3943-4a7d-a9b5-f4dea185ef43-openshift-service-ca\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.816231 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.816207    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1171b75e-3943-4a7d-a9b5-f4dea185ef43-maas-api-tls\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.816328 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.816245    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1171b75e-3943-4a7d-a9b5-f4dea185ef43-metrics-certs\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.822255 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.822229    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9jzw\" (UniqueName: \"kubernetes.io/projected/1171b75e-3943-4a7d-a9b5-f4dea185ef43-kube-api-access-v9jzw\") pod \"maas-api-5fbc95fb78-g6cqp\" (UID: \"1171b75e-3943-4a7d-a9b5-f4dea185ef43\") " pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:28.857311 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:28.857274    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:29.134360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:29.134333    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"]
Nov 02 18:53:29.137207 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:53:29.137178    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1171b75e_3943_4a7d_a9b5_f4dea185ef43.slice/crio-2d609522565fb12b628175c15951d519c188654a708a104838a4fc9d8fbbb804 WatchSource:0}: Error finding container 2d609522565fb12b628175c15951d519c188654a708a104838a4fc9d8fbbb804: Status 404 returned error can't find the container with id 2d609522565fb12b628175c15951d519c188654a708a104838a4fc9d8fbbb804
Nov 02 18:53:29.415196 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:29.415106    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp" event={"ID":"1171b75e-3943-4a7d-a9b5-f4dea185ef43","Type":"ContainerStarted","Data":"2d609522565fb12b628175c15951d519c188654a708a104838a4fc9d8fbbb804"}
Nov 02 18:53:30.427338 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:30.427298    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp" event={"ID":"1171b75e-3943-4a7d-a9b5-f4dea185ef43","Type":"ContainerStarted","Data":"84773f4f303e8f8cefeaefe8717f362ddee36c2ceb4ec15b48aeb390831f9736"}
Nov 02 18:53:30.427910 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:30.427370    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:30.446989 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:30.446919    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp" podStartSLOduration=2.023490022 podStartE2EDuration="2.446897029s" totalImagesPullingTime="423.407007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:30.444876095 +0000 UTC m=+4823.024285483" watchObservedRunningTime="2026-11-02 18:53:30.446897029 +0000 UTC m=+4823.026306416"
Nov 02 18:53:34.024236 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:34.024169    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"]
Nov 02 18:53:34.028117 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:34.028070    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-d6n4g"]
Nov 02 18:53:34.421927 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:34.421787    5042 scope.go:122] "RemoveContainer" containerID="38a5aa2512a34e9d30eee4de6010b3f5134666a28ae42d64c38ff2bed85d18c9"
Nov 02 18:53:34.435360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:34.435319    5042 scope.go:122] "RemoveContainer" containerID="4b54432a4639f3c24564bc043f6e7a50b29e5f05caefb4867f4894d409474a9b"
Nov 02 18:53:34.448213 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:34.448182    5042 scope.go:122] "RemoveContainer" containerID="774de9c8bd23582e292a50ba0222fd8bc34e4cafd155717f1e650a6ae0fcb993"
Nov 02 18:53:35.549214 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:35.549181    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7251c1e5-f22c-4d78-989d-e50bf4fb6c4f" path="/var/lib/kubelet/pods/7251c1e5-f22c-4d78-989d-e50bf4fb6c4f/volumes"
Nov 02 18:53:36.459766 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.459733    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fbc95fb78-g6cqp"
Nov 02 18:53:36.528868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.528447    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"]
Nov 02 18:53:36.528868 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.528697    5042 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt" podUID="b66490f7-e2bd-4e56-be55-e7390bc2ae5f" containerName="maas-api" containerID="cri-o://34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0" gracePeriod=30
Nov 02 18:53:36.873019 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.872990    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:36.968154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.968115    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs\") pod \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") "
Nov 02 18:53:36.968154 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.968160    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca\") pod \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") "
Nov 02 18:53:36.968439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.968218    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd\" (UniqueName: \"kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd\") pod \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") "
Nov 02 18:53:36.968439 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.968235    5042 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls\") pod \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\" (UID: \"b66490f7-e2bd-4e56-be55-e7390bc2ae5f\") "
Nov 02 18:53:36.968736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.968565    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca" pod "b66490f7-e2bd-4e56-be55-e7390bc2ae5f" (UID: "b66490f7-e2bd-4e56-be55-e7390bc2ae5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:53:36.968736 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.968713    5042 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-openshift-service-ca\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:36.970747 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.970722    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls" pod "b66490f7-e2bd-4e56-be55-e7390bc2ae5f" (UID: "b66490f7-e2bd-4e56-be55-e7390bc2ae5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:53:36.971100 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.971074    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs" pod "b66490f7-e2bd-4e56-be55-e7390bc2ae5f" (UID: "b66490f7-e2bd-4e56-be55-e7390bc2ae5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:53:36.971197 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:36.971098    5042 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd" pod "b66490f7-e2bd-4e56-be55-e7390bc2ae5f" (UID: "b66490f7-e2bd-4e56-be55-e7390bc2ae5f"). InnerVolumeSpecName "kube-api-access-fcrpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:53:37.069674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.069574    5042 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-metrics-certs\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:37.069674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.069609    5042 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcrpd\" (UniqueName: \"kubernetes.io/projected/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-kube-api-access-fcrpd\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:37.069674 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.069620    5042 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b66490f7-e2bd-4e56-be55-e7390bc2ae5f-maas-api-tls\") on node \"ip-10-0-135-5.us-east-2.compute.internal\" DevicePath \"\""
Nov 02 18:53:37.577369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.577331    5042 generic.go:356] "Generic (PLEG): container finished" podID="b66490f7-e2bd-4e56-be55-e7390bc2ae5f" containerID="34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0" exitCode=0
Nov 02 18:53:37.577570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.577381    5042 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"
Nov 02 18:53:37.577570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.577418    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt" event={"ID":"b66490f7-e2bd-4e56-be55-e7390bc2ae5f","Type":"ContainerDied","Data":"34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0"}
Nov 02 18:53:37.577570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.577501    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt" event={"ID":"b66490f7-e2bd-4e56-be55-e7390bc2ae5f","Type":"ContainerDied","Data":"003dca1c79133e076d26e36f006255402e4f2d9576832979f43f9ceb1f75b721"}
Nov 02 18:53:37.577570 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.577529    5042 scope.go:122] "RemoveContainer" containerID="34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0"
Nov 02 18:53:37.597928 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.597905    5042 scope.go:122] "RemoveContainer" containerID="34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0"
Nov 02 18:53:37.598284 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:53:37.598259    5042 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0\": container with ID starting with 34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0 not found: ID does not exist" containerID="34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0"
Nov 02 18:53:37.598369 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.598295    5042 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0"} err="failed to get container status \"34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0\": rpc error: code = NotFound desc = could not find container \"34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0\": container with ID starting with 34780b26e557757abeb4843d2fb07632b48d13d3128354cf21c46f77022dfea0 not found: ID does not exist"
Nov 02 18:53:37.603922 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.603898    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"]
Nov 02 18:53:37.607584 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:37.607560    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84f9c6994b-86mpt"]
Nov 02 18:53:39.541831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:39.541794    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66490f7-e2bd-4e56-be55-e7390bc2ae5f" path="/var/lib/kubelet/pods/b66490f7-e2bd-4e56-be55-e7390bc2ae5f/volumes"
Nov 02 18:53:44.022660 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:44.022616    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"]
Nov 02 18:53:44.024971 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:44.024943    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-z7s8z"]
Nov 02 18:53:45.542387 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:53:45.542352    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5be8a0-3611-490c-aefb-c888d65f5819" path="/var/lib/kubelet/pods/9d5be8a0-3611-490c-aefb-c888d65f5819/volumes"
Nov 02 18:53:48.542737 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:53:48.542702    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:54:18.538712 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:54:18.538676    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:54:23.650020 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:54:23.649966    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:54:23.650581 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:54:23.650080    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:54:34.509658 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:54:34.509624    5042 scope.go:122] "RemoveContainer" containerID="15451a7824f6c55dbf00a0ccd5414477fce70c8cbe7623bed00c462a998be467"
Nov 02 18:54:34.519030 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:54:34.519001    5042 scope.go:122] "RemoveContainer" containerID="6c0ac25d106b3204bef6c4a61d8fe157532934d46cd3f2d1aa426c7fa7ebfe60"
Nov 02 18:54:34.528232 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:54:34.528210    5042 scope.go:122] "RemoveContainer" containerID="0edc054771db1a35a4a8e995d5bac5fb93c4509194ccd8300bbbf9c2bf279ef3"
Nov 02 18:54:34.537370 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:54:34.537349    5042 scope.go:122] "RemoveContainer" containerID="ee5c0ddbfbd287283260896574868b551be8e82dc7b5595f633950eda081d3e4"
Nov 02 18:54:48.539536 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:54:48.539462    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:55:05.028874 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:05.027310    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"]
Nov 02 18:55:05.036708 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:05.036671    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ace16cc-rmcjx"]
Nov 02 18:55:05.542870 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:05.542817    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605a9cd0-5e77-4b4d-bc36-07f4ffe014e8" path="/var/lib/kubelet/pods/605a9cd0-5e77-4b4d-bc36-07f4ffe014e8/volumes"
Nov 02 18:55:18.538473 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:55:18.538438    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:55:34.578112 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:34.578074    5042 scope.go:122] "RemoveContainer" containerID="08a020fe98adaea71bd2717c59b7e9f21067fe3c959df23e0c109e74a9ccd632"
Nov 02 18:55:34.588911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:34.588884    5042 scope.go:122] "RemoveContainer" containerID="2a7079b7100cccbf2d43cd7834bf61a13067597a6c7c3f3b955e62a8f184ed41"
Nov 02 18:55:48.361069 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:48.361013    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:55:48.361653 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:48.361113    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:55:48.534978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:48.534063    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:55:48.534978 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:55:48.534205    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:55:48.555183 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:55:48.555023    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:56:14.022945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:56:14.022908    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"]
Nov 02 18:56:14.024982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:56:14.024956    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xs8gl"]
Nov 02 18:56:15.542342 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:56:15.542302    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f348999-4af2-4c94-81fd-6ee0f7384246" path="/var/lib/kubelet/pods/6f348999-4af2-4c94-81fd-6ee0f7384246/volumes"
Nov 02 18:56:18.539621 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:56:18.539539    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:56:34.622587 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:56:34.622551    5042 scope.go:122] "RemoveContainer" containerID="fa7651e40336cef554331ddad69cb5028d92f641cf35b1eea55db3d576eed8e2"
Nov 02 18:56:48.539033 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:56:48.538994    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:57:18.359032 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:18.358993    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:57:18.359545 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:18.359063    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:57:18.538487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:18.537971    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-xj2fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:57:18.538487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:18.538052    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-xj2fs" podUID="bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:57:18.576867 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:57:18.576374    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:57:34.644142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:34.644106    5042 scope.go:122] "RemoveContainer" containerID="f59ef923dc470a45bdfe39b577e10120e093b751315e6b3f59a5cfd41b9104ce"
Nov 02 18:57:34.653456 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:34.653428    5042 scope.go:122] "RemoveContainer" containerID="5bb50b1e09bbcf69c3ab5074c0e8a196a9ccf16b7bc15440bf067a4122314c50"
Nov 02 18:57:38.643009 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:38.642964    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-zws9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:57:38.643675 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:38.643637    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-zws9j" podUID="5557b2eb-4759-40b7-b6f2-c41494af6815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:57:48.371067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:48.370981    5042 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-65c448b547-dtrzt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:57:48.371067 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:48.371051    5042 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-65c448b547-dtrzt" podUID="5a334a6a-1a82-435e-9785-1047ec1c3487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:57:48.540471 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:57:48.540434    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:57:57.031061 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:57.031021    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"]
Nov 02 18:57:57.035108 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:57.035058    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-b8q5x"]
Nov 02 18:57:57.559157 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:57:57.558796    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547f8787-a024-4ce7-9ce1-b67d301e693b" path="/var/lib/kubelet/pods/547f8787-a024-4ce7-9ce1-b67d301e693b/volumes"
Nov 02 18:58:08.438077 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:08.438035    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fb64bd955-5m6ww_25e41b3c-1fd8-422b-aab6-a660c5b76e4e/authorino/0.log"
Nov 02 18:58:10.092781 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:10.092752    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:58:10.094057 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:10.094035    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:58:10.095669 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:10.095647    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:58:10.096831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:10.096804    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pswgg_269ca06c-b59b-496f-81aa-a08c525d0098/ovn-acl-logging/0.log"
Nov 02 18:58:13.023945 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:13.023912    5042 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"]
Nov 02 18:58:13.027147 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:13.027116    5042 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-xlcjr"]
Nov 02 18:58:13.543686 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:13.543650    5042 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fce1258-3b3e-43b7-829d-6863fa402353" path="/var/lib/kubelet/pods/0fce1258-3b3e-43b7-829d-6863fa402353/volumes"
Nov 02 18:58:18.539111 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:58:18.539076    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:58:20.768752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:20.768708    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/1.log"
Nov 02 18:58:22.483062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:22.483031    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/0.log"
Nov 02 18:58:23.042880 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:23.042852    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fbc95fb78-g6cqp_1171b75e-3943-4a7d-a9b5-f4dea185ef43/maas-api/0.log"
Nov 02 18:58:28.881785 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:28.881744    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fb64bd955-5m6ww_25e41b3c-1fd8-422b-aab6-a660c5b76e4e/authorino/0.log"
Nov 02 18:58:29.104956 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:29.104920    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-lx6js_06bdf526-bccd-4303-b9f5-ff3131a4ae96/manager/0.log"
Nov 02 18:58:29.302816 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:29.302785    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-k9mfj_d56437ee-40bb-439d-812f-1a55863eac69/manager/0.log"
Nov 02 18:58:29.679809 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:29.679725    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8c4p4_b5e8eadd-5714-470d-9adb-999d78f52191/manager/0.log"
Nov 02 18:58:30.089879 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:30.089834    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ljs7q_5488c1fe-6a3d-464a-bd15-2377b7740b9b/manager/0.log"
Nov 02 18:58:31.058801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:31.058771    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8_c6dd5948-ec52-4943-b15d-312a915f9b3f/istio-proxy/0.log"
Nov 02 18:58:31.819345 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:31.819313    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-dtrzt_5a334a6a-1a82-435e-9785-1047ec1c3487/istio-proxy/0.log"
Nov 02 18:58:33.188324 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:33.188292    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xj2fs_bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11/istio-proxy/0.log"
Nov 02 18:58:33.592552 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:33.592479    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-zws9j_5557b2eb-4759-40b7-b6f2-c41494af6815/istio-proxy/0.log"
Nov 02 18:58:33.781062 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:33.781029    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb_bc408cd5-d5f4-471f-b48d-1bc3b86eec74/istio-proxy/0.log"
Nov 02 18:58:34.326563 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:34.326533    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7d5754d7-v6znk_7adb34aa-6a54-4a58-8761-49b5de183bba/router/0.log"
Nov 02 18:58:35.962455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:35.962417    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4pb5f_9f804eed-b3bb-4531-81b5-6060fd90682b/main/0.log"
Nov 02 18:58:36.361849 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:36.361756    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8pn4zd_0b474fd7-9730-4acd-be17-6e9ae01a1e44/main/0.log"
Nov 02 18:58:36.556633 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:36.556607    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-c8zlh_23fa0458-aa25-47cf-9b62-00385b8646fd/main/0.log"
Nov 02 18:58:36.948549 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:36.948516    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-547hk_2803f60a-6320-44ec-8961-4a6f026d6a45/main/0.log"
Nov 02 18:58:48.539375 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:58:48.539336    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"
Nov 02 18:58:50.627801 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:50.627768    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b26bx_5188d247-bf34-437c-bde7-0abc73c312f5/global-pull-secret-syncer/0.log"
Nov 02 18:58:50.794600 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:50.794573    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-892gj_814ca1bd-35f4-4b88-9d95-f9e1cc47845b/connection-checker/0.log"
Nov 02 18:58:50.827480 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:50.827455    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bxzmw_a893c12e-173e-4d9c-8080-8fa9992d9396/connection-checker/0.log"
Nov 02 18:58:50.861370 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:50.861337    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nwfc8_f3c68c66-cc5c-4bc4-9937-691ece1b51b3/connection-checker/0.log"
Nov 02 18:58:51.079479 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:51.079430    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fh7z4_d806b75d-88ca-4e51-94c3-cd45fd8888db/konnectivity-agent/0.log"
Nov 02 18:58:51.114535 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:51.114505    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-5.us-east-2.compute.internal_4cabb9f1887dd6d87ae5b4fba788119d/haproxy/0.log"
Nov 02 18:58:58.167650 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:58.167608    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fb64bd955-5m6ww_25e41b3c-1fd8-422b-aab6-a660c5b76e4e/authorino/0.log"
Nov 02 18:58:58.568295 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:58.568263    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-lx6js_06bdf526-bccd-4303-b9f5-ff3131a4ae96/manager/0.log"
Nov 02 18:58:58.599931 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:58.599898    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-k9mfj_d56437ee-40bb-439d-812f-1a55863eac69/manager/0.log"
Nov 02 18:58:58.693353 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:58.693313    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-8c4p4_b5e8eadd-5714-470d-9adb-999d78f52191/manager/0.log"
Nov 02 18:58:58.820371 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:58:58.820285    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ljs7q_5488c1fe-6a3d-464a-bd15-2377b7740b9b/manager/0.log"
Nov 02 18:59:00.550007 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:00.549971    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-s9wxm_565006a5-a1dd-4383-8078-f99d66892a75/cluster-monitoring-operator/0.log"
Nov 02 18:59:00.929167 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:00.929070    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f68nl_470d6f34-0ce2-446f-b498-e411eb263932/node-exporter/0.log"
Nov 02 18:59:00.954481 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:00.954457    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f68nl_470d6f34-0ce2-446f-b498-e411eb263932/kube-rbac-proxy/0.log"
Nov 02 18:59:00.981283 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:00.981251    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f68nl_470d6f34-0ce2-446f-b498-e411eb263932/init-textfile/0.log"
Nov 02 18:59:01.345972 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.345945    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-vktlz_fa18f4d4-d9fa-4802-b90b-e7044d222b39/prometheus-operator-admission-webhook/0.log"
Nov 02 18:59:01.870911 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.870876    5042 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"]
Nov 02 18:59:01.871489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.871466    5042 cpu_manager.go:414] "containerMap: removing container" podUID="b66490f7-e2bd-4e56-be55-e7390bc2ae5f" containerName="maas-api"
Nov 02 18:59:01.871489 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.871489    5042 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66490f7-e2bd-4e56-be55-e7390bc2ae5f" containerName="maas-api"
Nov 02 18:59:01.871654 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.871625    5042 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66490f7-e2bd-4e56-be55-e7390bc2ae5f" containerName="maas-api"
Nov 02 18:59:01.874802 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.874779    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:01.883632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.883595    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6cx\"/\"openshift-service-ca.crt\""
Nov 02 18:59:01.883632 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.883615    5042 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nc6cx\"/\"default-dockercfg-qrpjr\""
Nov 02 18:59:01.883831 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.883705    5042 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nc6cx\"/\"kube-root-ca.crt\""
Nov 02 18:59:01.883963 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:01.883945    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"]
Nov 02 18:59:02.027350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.027311    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tsz2\" (UniqueName: \"kubernetes.io/projected/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-kube-api-access-2tsz2\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.027350 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.027353    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-sys\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.027597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.027377    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-lib-modules\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.027597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.027514    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-proc\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.027597 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.027533    5042 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-podres\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.128783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128700    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tsz2\" (UniqueName: \"kubernetes.io/projected/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-kube-api-access-2tsz2\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.128783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128742    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-sys\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.128783 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128767    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-lib-modules\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.129102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128883    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-sys\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.129102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128895    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-proc\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.129102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128933    5042 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-podres\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.129102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128936    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-lib-modules\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.129102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.128942    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-proc\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.129102 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.129021    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-podres\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.136683 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.136651    5042 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tsz2\" (UniqueName: \"kubernetes.io/projected/6dfb326d-a365-4ef0-b78f-4f61dfc37ec7-kube-api-access-2tsz2\") pod \"perf-node-gather-daemonset-dkjs9\" (UID: \"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7\") " pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.189425 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.189375    5042 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:02.464305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:02.464272    5042 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"]
Nov 02 18:59:02.467165 ip-10-0-135-5 kubenswrapper[5042]: W1102 18:59:02.467136    5042 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6dfb326d_a365_4ef0_b78f_4f61dfc37ec7.slice/crio-6e9b151ce2882551015b4df153b5715f92d039642fe3c75ca12c789acb4e1fd2 WatchSource:0}: Error finding container 6e9b151ce2882551015b4df153b5715f92d039642fe3c75ca12c789acb4e1fd2: Status 404 returned error can't find the container with id 6e9b151ce2882551015b4df153b5715f92d039642fe3c75ca12c789acb4e1fd2
Nov 02 18:59:03.024832 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.024786    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9" event={"ID":"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7","Type":"ContainerStarted","Data":"1cf5fdc70de1cec3d3d075ab596abead88d10b14c3a5441ec7889040b0201b84"}
Nov 02 18:59:03.025266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.024872    5042 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9" event={"ID":"6dfb326d-a365-4ef0-b78f-4f61dfc37ec7","Type":"ContainerStarted","Data":"6e9b151ce2882551015b4df153b5715f92d039642fe3c75ca12c789acb4e1fd2"}
Nov 02 18:59:03.025266 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.024981    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:03.042442 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.042388    5042 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9" podStartSLOduration=2.042374154 podStartE2EDuration="2.042374154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:59:03.040547836 +0000 UTC m=+5155.619957222" watchObservedRunningTime="2026-11-02 18:59:03.042374154 +0000 UTC m=+5155.621783539"
Nov 02 18:59:03.072063 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.072028    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-47kbn_0d4a9cee-81c7-46ce-b7c2-9c273dd7e6fc/networking-console-plugin/0.log"
Nov 02 18:59:03.846780 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.846748    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/1.log"
Nov 02 18:59:03.851854 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:03.851823    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-622zn_17ccbff4-2818-4e70-b2f4-a64c92f789e8/console-operator/2.log"
Nov 02 18:59:04.575381 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:04.575348    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-98bw2_6377c122-4ad5-4829-a2df-ed251a70d3bc/download-server/0.log"
Nov 02 18:59:04.599982 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:04.599947    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/0.log"
Nov 02 18:59:04.603671 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:04.603647    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/1.log"
Nov 02 18:59:05.370217 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:05.370183    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-4559l_d55feb08-7024-4795-9a42-7345567fe78b/volume-data-source-validator/0.log"
Nov 02 18:59:07.087487 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:07.087452    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gfg92_63048823-3df5-48fa-a65d-c1d789a319c9/dns/0.log"
Nov 02 18:59:07.112360 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:07.112330    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gfg92_63048823-3df5-48fa-a65d-c1d789a319c9/kube-rbac-proxy/0.log"
Nov 02 18:59:07.251773 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:07.251742    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzlmp_032f9f33-d723-46ad-9d89-45956596eec5/dns-node-resolver/0.log"
Nov 02 18:59:08.018466 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:08.018425    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bb6c9c46c-66dr9_417a9298-1e3c-46bb-965c-aab3c75b8728/registry/0.log"
Nov 02 18:59:08.043959 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:08.043932    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dv585_7e1fc379-cd57-4b14-be52-b3975a5f6514/node-ca/0.log"
Nov 02 18:59:08.613305 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:08.613268    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/0.log"
Nov 02 18:59:08.617210 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:08.617188    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/1.log"
Nov 02 18:59:09.041697 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:09.041670    5042 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nc6cx/perf-node-gather-daemonset-dkjs9"
Nov 02 18:59:09.390212 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:09.390121    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8_c6dd5948-ec52-4943-b15d-312a915f9b3f/istio-proxy/0.log"
Nov 02 18:59:10.115889 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:10.115856    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-dtrzt_5a334a6a-1a82-435e-9785-1047ec1c3487/istio-proxy/0.log"
Nov 02 18:59:10.359926 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:10.359894    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xj2fs_bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11/istio-proxy/0.log"
Nov 02 18:59:10.435652 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:10.435560    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-zws9j_5557b2eb-4759-40b7-b6f2-c41494af6815/istio-proxy/0.log"
Nov 02 18:59:10.462906 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:10.462875    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb_bc408cd5-d5f4-471f-b48d-1bc3b86eec74/istio-proxy/0.log"
Nov 02 18:59:10.563827 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:10.563791    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7d5754d7-v6znk_7adb34aa-6a54-4a58-8761-49b5de183bba/router/0.log"
Nov 02 18:59:13.038504 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:13.038463    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/0.log"
Nov 02 18:59:13.042853 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:13.042806    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-tfbt9_6f4e4902-3ee4-40f1-a1d5-a7720cf9d7a0/manager/1.log"
Nov 02 18:59:14.575979 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:14.575952    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gqgqg_1f693f58-c274-4b42-863d-7b4e73d8fb10/serve-healthcheck-canary/0.log"
Nov 02 18:59:15.068626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.068600    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8_c6dd5948-ec52-4943-b15d-312a915f9b3f/istio-proxy/0.log"
Nov 02 18:59:15.314444 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.314414    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-hk6xf_52a1ad24-d5c4-49cf-aa71-b92f14918879/insights-operator/0.log"
Nov 02 18:59:15.315752 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.315731    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-hk6xf_52a1ad24-d5c4-49cf-aa71-b92f14918879/insights-operator/1.log"
Nov 02 18:59:15.339817 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.339755    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hffvc_72ce72d3-fe26-4612-ac57-078f9f7bbfbe/kube-rbac-proxy/0.log"
Nov 02 18:59:15.364137 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.364112    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hffvc_72ce72d3-fe26-4612-ac57-078f9f7bbfbe/exporter/0.log"
Nov 02 18:59:15.390239 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.390208    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hffvc_72ce72d3-fe26-4612-ac57-078f9f7bbfbe/extractor/0.log"
Nov 02 18:59:15.430687 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.430652    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-dtrzt_5a334a6a-1a82-435e-9785-1047ec1c3487/istio-proxy/0.log"
Nov 02 18:59:15.576626 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.576592    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xj2fs_bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11/istio-proxy/0.log"
Nov 02 18:59:15.585091 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.585064    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-fjcwd-h6mhp_f02abe3a-c622-428b-9650-bdfd6ebf85fd/insights-gathering/0.log"
Nov 02 18:59:15.615135 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.615055    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-zws9j_5557b2eb-4759-40b7-b6f2-c41494af6815/istio-proxy/0.log"
Nov 02 18:59:15.618954 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.618933    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-rx2qf-5jt9h_8645d924-656d-4790-a0d1-511c42b60e2b/insights-gathering/0.log"
Nov 02 18:59:15.633598 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.633573    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb_bc408cd5-d5f4-471f-b48d-1bc3b86eec74/istio-proxy/0.log"
Nov 02 18:59:15.699142 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:15.699112    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7d5754d7-v6znk_7adb34aa-6a54-4a58-8761-49b5de183bba/router/0.log"
Nov 02 18:59:17.253321 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:17.253292    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68dd6f4fc-p5bh8_c6dd5948-ec52-4943-b15d-312a915f9b3f/istio-proxy/0.log"
Nov 02 18:59:17.546899 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:17.546795    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-dtrzt_5a334a6a-1a82-435e-9785-1047ec1c3487/istio-proxy/0.log"
Nov 02 18:59:17.711397 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:17.711363    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-xj2fs_bb1d62b5-27ee-4e99-9cb7-5ee8caebfd11/istio-proxy/0.log"
Nov 02 18:59:17.746455 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:17.746422    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-65c448b547-zws9j_5557b2eb-4759-40b7-b6f2-c41494af6815/istio-proxy/0.log"
Nov 02 18:59:17.761323 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:17.761299    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-69fd6b667f-n4flb_bc408cd5-d5f4-471f-b48d-1bc3b86eec74/istio-proxy/0.log"
Nov 02 18:59:17.848814 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:17.848733    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7d5754d7-v6znk_7adb34aa-6a54-4a58-8761-49b5de183bba/router/0.log"
Nov 02 18:59:18.345478 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:18.345437    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-789486f7c8-sx6g4_2e8003ad-c2d2-483b-8ccf-d48aadd55aed/manager/0.log"
Nov 02 18:59:18.364992 ip-10-0-135-5 kubenswrapper[5042]: I1102 18:59:18.364962    5042 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-789486f7c8-sx6g4_2e8003ad-c2d2-483b-8ccf-d48aadd55aed/copy-manifests/0.log"
Nov 02 18:59:18.538529 ip-10-0-135-5 kubenswrapper[5042]: E1102 18:59:18.538495    5042 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hffvc" containerName="extractor"